What is the point of

select_filter

? Isn't

select

a kind of filtering mechanism by definition? Why would I need a filter for a filter? 😅

I find it useful when using environment vars to filter a list down, like when I'm parallelizing a tap You can setup all the specefic fields you want / don't want in select and then just use select filter to pick which streams you want. I was surprised to find a place I wanted the use case but that's the only one for me! It is super useful 😄

yes i use it for the same. in airflow we run one stream per task and use select_filter to choose the stream. but the select is the backstop, we can never select more than what is configured this way. so in the select you could have it narrowed to N columns on a stream, but your select filter does not need to be that specific, just

stream.*

which is good for keeping configuration in Meltano only.

Also:

select_filter

is applied to the provided catalog.

Thanks everyone 🙂

Will you all take a look at the doc updates in https://github.com/meltano/meltano/pull/9434 ?

The comparison table is good, thanks.

Reading through it I only have comments but I don't have the time to give productive feedback 😕

FYI, I updated that PR to remove the

stream

==

stream.*

behavior. I think that'd be better for v4. What that PR now does is a) clarify the docs a bunch and b) automatically includes the stream when a user only does

stream.property