Where can I find the catalog for a certain tap?

See http://meltano.com/docs/integration.html#extractor-catalog-generation

If you'd like to manually inspect the generated catalog for debugging purposes, you can dump it to STDOUT or a file using the

--dump=catalog

option on

meltano invoke

or

meltano elt

.

Specifically I want to be able to adjust datatypes within the catalog, is there a way to edit the catalog that is being used for the ELT

There is! https://meltano.com/docs/integration.html#overriding-schemas https://meltano.com/docs/plugins.html#schema-extra

Along these same lines, I’m trying to create a pipeline using

tap-mysql

and

target-redshift

. I have multiple databases in mysql and when I examine the catalog via

meltano select tap-mysql --list --all

, it looks like they’re getting captured in the format

database_name-table_name.field_name

. However,

target-redshift

doesn’t seem to like that and throws an

Exception writing records', PostgresError('Identifier must only contain lower case letters, numbers, underscores, or dollar signs.

Am I correct in assuming that I need to override the schema manually to reformat those data sources and if so, would the methods linked in the docs above be the appropriate way to approach that?

That sounds like the target-redshift version of this error on target-postgres: https://meltano.com/plugins/loaders/postgres.html#error-psycopg2-programmingerror-syntax-error-at-or-near. The suggestion is the same: give the

transferwise

variant a try instead of `datamill-co`: https://github.com/transferwise/pipelinewise-target-redshift

Cool, all that makes sense and tracks with what I’ve seen on my end too. I’ve tried the transferwise version (assuming that it should play nice with tap-mysql which also seems to be transferwise under the hood), but I get:

ELT could not be completed: Cannot start loader: Executable 'pipelinewise-target-redshift' could not be found. Loader 'pipelinewise-target-redshift' may not have been installed yet using `meltano install loader pipelinewise-target-redshift`, or the executable name may be incorrect.

I’ve run that included

meltano install

command (which seems to have worked fine) as well, but still no success. Any chance there are special tricks to the config on this one?

Can you share your

meltano.yml

plugin definition?

interesting! I started out using docker but have shifted away from it for the moment so I didn’t have to keep relying on verbose

docker run

commands everytime I wanted to try something 🙂

As long as you either go all-Docker or all-local, you're good, but don't mix!

totally - I’m planning on all-local for testing at this point and then potentially containerizing in production down the road

All right, good. So you're running

meltano install

and

meltano elt

both without Docker? Because then that issue isn't it 🙂

in the meantime, here’s the relevant

meltano.yml

chunk:

- name: pipelinewise-target-redshift
    namespace: pipelinewise_target_redshift
    pip_url: pipelinewise-target-redshift
    executable: pipelinewise-target-redshift
    config:
      host: REDACTED
      port: REDACTED
      user: REDACTED
      password: REDACTED
      dbname: REDACTED
      aws_access_key_id: REDACTED
      aws_secret_access_key: REDACTED
      s3_bucket: REDACTED
      default_target_schema: REDACTED

Try

executable: target-redshift

ah! thanks - let me try that

My pleasure, glad we figured it out!