Hi, I'm trying to use --catalog to provide the catalog manually and why I'm trying to figure out is if I want the select rules to be applied using this schema how can it be done? because I found that this rules are being applied only when the discovery is being done. Any suggestions?

@or_barda If you're manually providing a catalog file, Meltano will not apply any select, metadata, or schema rules, because

meltano elt --catalog

suggests that that exact catalog should be passed to the extractor. Select filters are the exception, because they can also be specified on the CLI:

meltano elt --catalog=... --select=... --exclude=...

See https://meltano.com/docs/plugins.html#catalog-extra:

If a catalog path is not set, the catalog will be generated on the fly by running the extractor in discovery mode

and applying the schema, selection, and metadata rules to the discovered file.

Selection filter rules are always applied to manually provided catalogs as well as discovered ones.

And https://meltano.com/docs/plugins.html#select-extra:

These rules are not applied when a catalog is provided manually.

I'm using the same tap in a different way and didn't want to replicate the catalog file

@or_barda Between the different uses of the tap, are you only varying the selected streams/properties? In that case, I'd suggest marking all of the streams/props you're interested in as

selected: true

in the catalog file, and using

select_filter

in

meltano.yml

or

--select

/`--exclude` on

meltano elt

to select a specific subset for each pipeline run

Interesting, I'll try it right now

Great, happy to help!