tap-jira

again (but maybe general question): I am regularly struggling with how to

select

streams/attributes(nested) with wild cards. Jira issues stream contains

fields

attribute, which contains a huge and nested structure, including hundreds of custom fields. I need to select just a small subset of nested fields, e.g.

assignee

. I tried many variants but nothing works, assignee is never persisted in target table (in Snowflake). I tested:

select:
        - issues.fields.assignee*
        - issues.fields.assignee.*

I also tried to flatten attributes:

config:
        flattening_enabled: 'True'
        flattening_max_depth: 1

and select:

select:
        - issues.fields__assignee*

does not work. Any advice? Does this work the same in all taps or can a tap affect how this behaves?

@jan_soubusta Are these fields custom or are they known to the tap beforehand (i.e. discoverable)?

They are IMO discoverable. When I select

issues.*

, they are materialized to the target table. When I use the flattening, each custom field is a separate column, so the table is extremely wide, which does not help to performance, at least in some databases 😉 That and the fact that I don't want them at all are reasons why I would like to filter them out.

Does this help? https://github.com/edgarrmondragon/tap-zendesk/?tab=readme-ov-file#selected-custom-fields

there are hundreds of them. Defining all of them in this way ... I don't want to do it 😉

Right, gotcha. How about

select:
        - issues.*
        - !issues.fields.*
        - issues.fields.assignee
        - issues.fields.assignee.*

?

I try it, thanks

Yeah, let me know if that works. Might be worth including the solution you find whenever https://github.com/meltano/meltano/issues/3024 gets done.

Does not work, custom fields are still reported.

You too!