can we write a

select

rule (e.g. tablename.*) for filtering nested jsons. for example I have a json in the source:

col: {key1: {"id": ..., "name": ...}, key2: [], }

can we do something like this:

select:
- table.col.key1.name
- table.col.key2

p.s. doesn't have be done in the select part, is some kind of definition possible?

At least taps built on the Singer SDK should support that. You can check if your

select

configuration is affecting the catalog as expected by running

meltano select <your tap> --select --list

.

p.s. doesn’t have be done in the select part, is some kind of definition possible?

Can you expand on what you mean by this? You want to define selection outside of

select

?

No, I don't specifically want to define selection outside of

select

. I meant it doesn't matter where I define it. If there are some other way to configure this type of feature.

meltano select tap-jira --list --all | grep issues.fields

It seems to get some of the keys from the

fields

jsonb. I'm guessing these are default keys, it didn't connect to the data-source and look up for these fields because in reality there are way more keys than these 4.

[selected ] issues.fields
[selected ] issues.fields.attachment
[selected ] issues.fields.created
[selected ] issues.fields.lastViewed
[selected ] issues.fields.updated
[selected ] issues.fieldsToInclude

I’m guessing these are default keys, it didn’t connect to the data-source and look up for these fields because in reality there are way more keys than these 4.

That’s correct. Most taps for API sources hardcode the catalog fields and the way to add more fields is to update the schemas: https://github.com/singer-io/tap-jira/blob/ff684a8721bf0049034a3f897f98f7fd55e58313/tap_jira/schemas/issues.json#L134-L174