I’ve been spinning why wheels on this issue for a few hours now so I figure it’s time to phone a friend We have a new table that needs loading via meltano, so I did what I normally do and add it in

meltano.yml

, run a

meltano elt

to make sure things load properly, then add it to the scheduled

meltano elt

command in prod that corresponds to our desired scheduling for updates to the new table. We’re using airflow with directly configured jobs to run these commands (each with a different

state_id

to distinguish them). This all works fine, so I let it run for a few days. Then I noticed that jobs being triggered by airflow were running fine, but not actually updating the table. Further investigation revealed that the new table had not gotten added to the job’s state JSON, which seemed odd. Grepping in the corresponding

tap.properties.json

also indicated that new table wasn’t listed there either. I know that running

meltano etl

should automatically trigger discovery and updating of

tap.properties.json

but somehow that doesn’t seem to be happening here? More oddly, everything continues to work just fine when I spin up a local version of the stack (via docker compose) and run the exact same command. There’s gotta be something different about production that’s not updating the tap properties correctly, but I can’t think of what it might be. Has anyone else ever run into this kind of thing or have any ideas?

Sounds like the cached catalog is what's causing your issue in Prod. What I normally do to wipe the cached catalog is to run a re-install of the tap https://github.com/meltano/meltano/issues/2856

ah, thanks for the pointer to that issue @christoph! I had a feeling something like this might be at play, but haven’t yet tried tinkering with the catalog yet. I’ll do some exploring along these lines and report back

am I correct in thinking that

meltano install

is the command you’re using to reinstall the taps when you ran into this before?

Yup. That's how I currently wipe the cached catalog for any tap that might dynamically change its streams in the catalog based on changes in the source.

Ah! Good to know, I didn't do that and will give that a shot. Thanks!

What is the tap that you are using btw? A public tap from the hub, or a custom tap?

It's pipelinewise-tap-mysql, but it's a fork

pipelinewise-tap-mysql, but it's a fork

I'm using that myself and actually have found it to be very reliable. I'll try and do a reproduction of the catalog caching issue using that tap ...

@nick_hamlin Does the new table show up when you run

meltano select tap-mysql --list

inside the container? Or

meltano select tap-mysql --list --all

?

ok, a few observations around `meltano select tap-mysql --list`: 1. When I run this command (and others) both locally and in prod, there’s a few minute long delay when nothing is apparently happening before I get the error about

discovery.yml

described here: https://meltano.slack.com/archives/C01TCRBBJD7/p1659995696908979 2. Running that command on our meltano image running on a local docker stack returns results that include the new table I’ve been trying to add. This is what I’d have expected, since I’m able to load results into it when running locally. 3. Running that command on the same image running in prod does NOT return results for the new table (which sorta also makes sense since that’s where it’s not working). What’s mystifying to me is why those two would be different when they’re using the same image. That implies to me that is has something to do with data stored in the underlying meltano database, since (I think?) that’s the only thing that’s capable of persisting data and is different between the two?

@nick_hamlin That discovery error was fixed in 2.5.0, are you able to update? What you’re describing does sound like a catalog caching issue, but I’m not aware of any caching there that would get baked into the Docker image 😕 On the prod env where

meltano select tap-mysql --list --all

doesn’t include the new table, does

meltano invoke tap-mysql --discover

have it? That just runs discovery mode directly, skipping any possible caching

I’ve updated the image to 2.4.0, but haven’t yet tried 2.5.0 - I can definitely give that a shot though. I’ll also see what happens with that second command, but this has also given me another idea of what might be happening to test. I’m wondering if the prod user doesn’t have explicit access to the new table but my local user does? I’ve got a ping into DevOps to take a look on that front too

@nick_hamlin Can you file an issue for that suggestion please?

sure can!

It does make me wonder if there might be an opportunity for meltano to

return more expressive errors in the future in situations where a data

source is indicated in

meltano.yml

but for which the user doesn’t actually have access permissions?

It certainly sounds like an interesting feature. Especially as the behaviour you experienced probably violates the principle of least surprise.

where a data source is indicated in

meltano.yml

but for which the user doesn’t actually have access permissions?

Does this mean that you had the table explicitly listed in the

select

rule, not just as

*.*

? I think it’d make sense to warn the user “this rule didn’t match any streams, make sure you don’t have a typo or that the stream exists (or you have access)

Yep, that’s exactly right - we have all tables for that tap explicitly listed (some additionally have specific columns enumerated, while others follow the format

database-table.*

)