What is the best way to execute meltano run with select or exclude, they do not seem to be supported flags on the run command. I have a pipeline where occasionally one or 2 of the streams will fail, i’d like to separate the streams into individual jobs so that one stream does not cause the others to fail or lose state.

My approach to that would be to define a virtual plugin using the inherit_from option, then include only the one or two problematic tables in the select rules for that virtual plugin. Then you can execute one pipeline using the "real" extractor and one using the virtual one.

interesting thank you

select_filter works as well!

Interesting. That can be used to exclude the problematic tables so they don't mess everything else up; a second pipeline would still be required to move the data in the troublesome tables though, right? Or are you suggesting having one pipeline and changing the select rules between runs?

the virtual plugin worked btw thank you. Unfortunately when I tried to use select-filter with a job and/or a run it does not work because the flag is not supported. Only with meltano elt does it work.

@datamongus https://docs.meltano.com/concepts/plugins#select_filter-extra setting the env variable should work!

Agreed. Thanks for the clarification!

oh interesting, thank you

https://docs.meltano.com/concepts/plugins#select_filter-extra setting the env variable should work!

Env variables is how I manage the select filters for my jobs. Works well for my case.