How do selection patterns work for taps with dynamic stream discovery (e.g.

pipelinewise tap-postgres

)? I can't seem to be able to get

meltano select tap-postgres --list --all

to print out an up-to-date list of entities.

meltano select

seems to be giving me an outdated list. Running

meltano invoke tap-postgres --discover

shows me the correct list of entities that reflect the current state of my database.

Hi @christoph. Yeah, as you discovered it's a missing feature: https://github.com/meltano/meltano/issues/2856 There are some proposals in the issue, though

elt

has to be replaced with

run

since that's the recommended command now. It's a more generic command, so probably

--with_discover

would instead need to be

--no-cache

or similar. Wdyt? Would you favor any of those, or a totally different approach?

Thanks @edgar_ramirez_mondragon! Having had a quick high-level read of AJ's proposal, I do favour Option B - purely because that is how I expect the cache refresh to work, i.e. as a user of the interactive meltano invocation during pipeline development, I want the cache behaviour to be seamless. (My personal focus is always implicitly on 'Developer Experience', because I want to keep developers happy!) Whilst I want to be aware that the cache is there, I certainly would not want to have to deal with the additional brainpower associated with an additional explicit command to run (i.e.

meltano, clear my cache first

) before I can run the actual task that I intended to perform (i.e.

meltano, show me all the entities that you discovered "right now" in the catalog

). I don't have too much of an opinion on the syntax or semantics of the actual implementation though. I think that part is something that most people will get used to. I do think that

--with-discover

is better, because it is way less ambiguous than

--no-cache

. Another option I can think of would be

--refresh-catalog

NOTE: Whichever option ends up getting implemented, I think it would need to be added to all commands that deal with the catalog. The commands that I ran into that deal with the catalog are

run, (elt), select

@christoph Awesome, thank you for sharing your thoughts on the subject!