Hello! I noticed that taps that have already been installed within a project don't update their properties when installing that tap again. The tap properties within

/.meltano/run

are being cached and not updated if there was a change within the tap. Is this intended by default?

@michael_cooper What command are you seeing use an outdated (cached) catalog?

meltano select

or

meltano elt

?

It might just be easier to type out what I'm doing step-by-step:

# install all taps
1. meltano install
2. meltano invoke tap-custom-tap

# update tap-custom-tap repo with some new schema properties
3. meltano install tap-custom-tap
4. meltano invoke tap-custom-tap

When I do that, I get an error because the schema/properties have been updated, but it's trying to run on a cached version of that tap's properties.

If you run

meltano --log-level=debug invoke tap-custom-tap

, do you see "Using cached catalog file" printed? If you delete

.meltano/run/tap-custom-tap/tap.properties.cache_key

, does a new catalog get generated?

It regenerates if I delete

.meltano/run/tap-custom-tap/tap.properties.cache_key

All right. Silver lining is that this caching is not used with

meltano elt

(https://gitlab.com/meltano/meltano/-/issues/2460), so you shouldn't see the issue when you're actually running pipelines 🙂 We should improve the cache invalidation logic though. Can you please file an issue?

Here's the debug of the first lines after an install of that tap:

[2021-03-04 21:28:53,618] [1|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x7f510102d160>@sqlite:////projects/.meltano/meltano.db
[2021-03-04 21:28:54,642] [1|MainThread|root] [DEBUG] Created configuration at /projects/.meltano/run/tap-s3-toast/tap.config.json
[2021-03-04 21:28:54,643] [1|MainThread|root] [DEBUG] Could not find tap.properties.json in /projects/.meltano/extractors/tap-s3-toast/tap.properties.json, skipping.
[2021-03-04 21:28:54,643] [1|MainThread|root] [DEBUG] Could not find tap.properties.cache_key in /projects/.meltano/extractors/tap-s3-toast/tap.properties.cache_key, skipping.
[2021-03-04 21:28:54,643] [1|MainThread|root] [DEBUG] Could not find state.json in /projects/.meltano/extractors/tap-s3-toast/state.json, skipping.
[2021-03-04 21:28:54,643] [1|MainThread|meltano.core.plugin.singer.tap] [DEBUG] Using cached catalog file

The cache is only used on

meltano invoke

and

meltano select

, because Meltano will look in

.meltano/run/<tap name>

for files in both of those cases, while

meltano elt

will look in

.meltano/run/elt/<UUID>

, which is new on every run, so there's never any cache to be picked up

Great! Still want me to submit the issue then?

Please do! Because we should fix the bug in

meltano invoke/select

and actually invalidate on

meltano install

And sorry since I've never submitted an issue for Meltano, should I use the

bug

,

feature_proposal

, or some other template for the issue?

"bugs" please!