Anyone know when how schemas are being created I thought tha Meltano #troubleshooting

Anyone know when/how schemas are being created? ...

andrew_stewart

04/02/2021, 11:15 PM

Anyone know when/how schemas are being created? I thought that by default meltano (or the singer target?) would go and create new schemas named after each extractor .. ie

tap_mongodb

or whatever.. but for some reason everything is just getting dumped into

public

. I’ve tried playing around with the

postgres_schema

config on

target-postgres

as well as

load_schema

on the individual extractors, but I keep getting errors about how the schemas don’t exist. Am I missing something or is this kind of strange?

andrew_stewart

04/03/2021, 12:28 AM

Interesting. It looks like for

target-postgres

the

meltano

variant supports dynamically creating schemas, while the

datamill-co

variant does not.

taylor

04/03/2021, 1:07 AM

I noticed this with the datamill snowflake target. it's their expected behavior but I'm not a fan of it. it's a better ux I think to autocreate the schema

andrew_stewart

04/03/2021, 8:38 PM

Yeah. I’m trying to figure out if there’s an natural way to list differences between variants via discovery.

andrew_stewart

04/03/2021, 9:42 PM

Just thinking out loud here as I skim through the cli and discovery.yml…

meltano discover loaders

(for example) conveniently lists all the registered variants, ala:

target-postgres, variants: datamill-co (default), transferwise, meltano

Then as far as I can tell something like

meltano config target-postgres list

doesn’t let you specify a particular variant. I wonder if there’d be some utility in having a command like

meltano info PLUGIN_NAME [--variant=VARIANT_NAME]

which outputs some basic info on plugin, as well as any “variant notes”. For example, variants listed in discovery.yml could include some curated notes like:

Copy code

- name: target-postgres
    label: PostgreSQL
    description: PostgreSQL database loader
    namespace: target_postgres
    dialect: postgres
    target_schema: $TARGET_POSTGRES_SCHEMA
    variants:
      - name: datamill-co
        notes:
          - label: Automatic schema generate
            value: No
          - label: foo
            value: bar
        ...
      - name: meltano
        original: true
        notes:
          - label: Automatic schema generate
            value: Yes
          - label: foo
            value: bar

Maybe something like this already exists I’m not aware of though.

andrew_stewart

04/03/2021, 9:43 PM

(if not, I’m happy to write up a feature request for deliberation)

2 Views

Open in Slack

Previous Next