Any help would be greatly appreciated. I have this configuration in the screenshot and

meltano config tap-mysql --discover > database-name-catalog.json

populates the JSON file without issues but when I run

meltano invoke tap-mysql

or

meltano config tap-mysql test

I get

Plugin configuration is invalid
No RECORD message received

And when I do

meltano select tap-mysql --list --all

everything is excluded

Maybe your

select

criteria is incorrect?

If it is, I haven't been able to figure out how. It is the the tap_stream_id from the catalog and an * Running

meltano select tap-mysql --list -all

shows the same table name with the database name as a prefix and every column as excluded. I even tried the table_name from the catalog.json instead of the tap_stream_id and still nothing. No matter what I have tried, everything is excluded and nothing works.

Try the select without the quotes?

😞 Tried that

It might be that you're trying to use

select

while overriding the

catalog

file - I assume nothing is selected in

database-name-catalog.json

? Try removing that and it should start respecting your

select

criteria.

So if selected by default is false, the select in the

meltano.yml

is ignored. It now shows as selected. I don't have time test thoroughly but I suspect you have saved the day.

I think it's more like if provide your own catalog file, you are taking control of catalog generation away from Meltano, and so you don't get the benefits of having the

schema

,

select

and

metadata

rules applied from your

meltano.yml

. https://docs.meltano.com/concepts/plugins/#catalog-extra

If I remove the catalog, I get `Stream ``summary`` was not found in the catalog`.

summary

being the one table I'm trying to transport

Do you see anything of note running

meltano invoke tap-mysql

now?

config test

tries to be smart when forming the

Plugin configuration is invalid

message, but it's sort of dependent on the tap. Generally, I would debug deeper issues like this with

invoke

and inspect the output.

invoke

doesn't show anything of note as far as I can tell

Do you see the

summary

stream and properties in

meltano select tap-mysql --list --all

?

If I include the catalog and the select properties, yes. summary shows as selected

Sure, and if you remove your

catalog

override and keep just the

select

? Do you get the same behaviour for

select

and

invoke

commands?

HOLY SMOKES... invoke shows rows and rows of data

So, to confirm - you are no longer overriding

catalog

and still get the

Plugin configuration is invalid

error from

config test

, but you are seeing

RECORD

messages out from

invoke

now?

yes

Alright, sounds like

config test

isn't respecting the

select

criteria for some reason. Do you have a sec to open an issue with some info? https://github.com/meltano/meltano/issues/new?assignees=meltano%2Fengineering&labels=kind%2FBug%2Cvaluestream%2FMeltano&projects=&template=bug.yml&title=bug%3A+%3Ctitle%3E Otherwise, I can try to piece one together from this thread later (I don't have a reproducible example though).

I don't right this second, but would be happy to in a few hours

Yeah, no rush at all. Thanks 😄

thank you; you are my hero

Haha, you're welcome! Glad we got there in the end.

I wonder if it is unrelated to

select

and is something between

test

and

tap-mysql

then. Like an expected configuration that isn't clear or that I didn't include

I'm pretty sure

config test

and

invoke

run through the same code path internally to invoke the plugin... Are you running on Windows?

Mac but running in the meltano docker container

Cheers, I'll follow along 😁

thanks again for the help