Meltano run is caching my catalog, I think for

elt

the catalog was always regenerated with discovery (unless you explicitly gave a catalog.json)

5 3262Z^[[0m [^[[32m^[[1mdebug    ^[[0m] ^[[1mCreated configuration at /home/visch/git/testcacheissue/.meltano/run/tap-postgres-testing/tap.cb7b16ce-72bc-43b0-8165-3d65ea79676a.config.json^[[0m
    4 3580Z^[[0m [^[[32m^[[1mdebug    ^[[0m] ^[[1mCould not find tap.properties.json in /home/visch/git/testcacheissue/.meltano/extractors/tap-postgres-testing/tap.properties.json, skipping.^[[0m
    3 3794Z^[[0m [^[[32m^[[1mdebug    ^[[0m] ^[[1mCould not find tap.properties.cache_key in /home/visch/git/testcacheissue/.meltano/extractors/tap-postgres-testing/tap.properties.cache_key, skipping.^[[0m
    2 3968Z^[[0m [^[[32m^[[1mdebug    ^[[0m] ^[[1mCould not find state.json in /home/visch/git/testcacheissue/.meltano/extractors/tap-postgres-testing/state.json, skipping.^[[0m
    1 7774Z^[[0m [^[[32m^[[1minfo     ^[[0m] ^[[1mFound state from 2022-07-28 05:17:47.411006.^[[0m
718   9285Z^[[0m [^[[32m^[[1mdebug    ^[[0m] ^[[1mUsing cached catalog file^[[0m

I'd give cleaner logs but I'm not sure of an easy way to do so Maybe someone can point me to a logging.yaml

This feels like the same caching issue I've had a few times. My understanding is that meltano caches the tap catalog and doesn't refresh it until/unless something changes in the config that would require it. Which is convenient... until it's not. Anyway, I also find the log output confusing. I don't think the cache files mentioned matter when using

meltano run

. To fix this I usually have to remove .meltano/run/<tapname>/tap.properties.cache_key.

My understanding was that happens, but only for

invoke

(still annoying I agree) I know there's an issue in for it. (It'd be so much better if it was in the "main" logs if we were / weren't using a cache as it's a guessing game for me otherwise so one step is to delete the .meltano/run dir) But for

elt

the catalog is refreshed every run (so I thought), and I think the same should be true for

run

. In my case my

select

even changed from the last run so there's a cache issue as well but I really don't care about the cache for this use case.

I've run into the cached catalog issue a couple times with

meltano run

with pipelines that have otherwise been running fine for a long time, so it's definitely not just an invoke problem. We can add this to the list of things that's not really meltano's fault - when the table schema in the source DB changes meltano has no way to know, so it has no way to know it should refresh the catalog. 😕

I'm hitting a cached catalog issue which is preventing new columns in csv files from being passed to target-postgres so they can be added to the table. I can manually resolve the issue by deleting the

.meltano\run\tap-csv\tap.properties.json

file, but I would love to be able to set something in my meltano.yml so I don't have to manually delete the cache. Is that possible?

Hi, @chrish. There are a few proposed fixes to this issue. The simplest is this proposal: feature: Add a `--refresh` option for `meltano select` to refresh a new copy of the discovered catalog · Issue #6919 · meltano/meltano (github.com)

Thanks @aaronsteers - I'll keep that in mind in case I get some time. It sounds like an interesting feature to work on.

-e also has the same issue I'm almost certain, but maybe it refreshed your catalog from the one change