Hello Team, Need help debugging this ELT operation. It shows extraction as completed but I do not see any changes/tables in the target database. Note: I'm running everything on docker: meltano, tap-mysql (MySQL DB as container), target-postgres (Postgres DB as container)

$ meltano elt tap-mysql target-postgres --job_id=mysql_to_postges_el

meltano | Running extract & load... meltano | A catalog file was found, but it will be ignored as the extractor does not advertise the

catalog

or

properties

capability target-postgres | /project/.meltano/loaders/target-postgres/venv/lib/python3.6/site-packages/psycopg2/__init__.py144 UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>. target-postgres | """) tap-mysql | INFO Server Parameters: version: 8.0.21, wait_timeout: 2700, innodb_lock_wait_timeout: 2700, max_allowed_packet: 67108864, interactive_timeout: 28800 tap-mysql | INFO Server SSL Parameters (blank means SSL is not active): [ssl_version: ], [ssl_cipher: ] tap-mysql | INFO No properties were selected meltano | Extract & load complete! meltano | Transformation skipped

@harsha_aleti The issue appears to be that your

tap-mysql

custom plugin definition does not list

catalog

or

properties

under

capabilities

, so the generated catalog/properties file is never passed on to the tap, per these log messages:

A catalog file was found, but it will be ignored as the extractor does not advertise the

catalog

or

properties

capability

INFO No properties were selected

Per https://github.com/singer-io/tap-mysql#sync-mode,

tap-mysql

supports the

properties

option, so you'll want to add it to

capabilities

in

meltano.yml

and try again.

Hey Douwe, Thank you so much for the quick response. I have added catalog and properties to `meltano.yml`file. But, I'm still getting the following error:

$ meltano --log-level=debug elt tap-mysql target-postgres

The error message is coming from here: https://github.com/singer-io/tap-mysql/blob/master/tap_mysql/__init__.py#L697 It appears that

tap-mysql

expects a

replication-method

metadata key with value

INCREMENTAL

,

LOG_BASED

or

FULL_TABLE

for each stream in the catalog, as well as `replication-id`: https://github.com/singer-io/tap-mysql#replication-methods-and-state-file You can set these using the

metadata

key in

meltano.yml

: https://meltano.com/docs/plugins.html#metadata-extra

I'm still running into the same error as before despite adding the metadata in meltano.yml

@harsha_aleti Can you dump the generated catalog with

meltano elt ... --dump=catalog

so that we can see if every stream actually has a

replication-method

set?

message has been deleted

Not quite what I had in mind 😄 Try this:

metadata:
    '*':
      replication-method: FULL_TABLE

$ meltano elt tap-mysql target-postgres --dump=catalog

Could not dump catalog: Applying catalog rules failed: catalog file is missing.

@harsha_aleti Wanna jump on a quick Zoom call so I can help you get to the bottom of this?

That will be awesome. Really appreciate if you can take time to quickly go over it.