Sorry, deleted the msg above. Core issue is: when I add a variant of a plugin with a different

pip_url

(eg running postgres wise for some taps, and meltanolabs version for others due to a few fixes still pending), then the variant doesn't seem to get installed correctly in docker image build. If I follow the guidance here, it works fine locally issuing meltano commands in the console : https://docs.meltano.com/guide/plugin-management#multiple-variants When I build the docker image for deployment, it no longer works. Dockerfile in thread above. Crucial log line below, the command was

meltano run tap-spreadsheets-anywhere target-postgres--meltanolabs

2023-07-12T11:28:05.967818Z [info     ]    * Required key is missing from config: [dbname] cmd_type=elb consumer=True name=target-postgres--meltanolabs producer=False stdio=stderr string_id=target-postgres--meltanolabs

it looks like the variant is reusing the env of the original plugin (as in simple inheritance, assuming the same pip_url). For instance, the

wise

variant uses

dbname

but the

meltanolabs

variant uses

database

. Edit: Also if I

meltano install --clean

locally, then a local run now fails as above.

Could you please share your

meltano.yml

? Then I can try to reproduce the issue 🤷

so when you inherit_from you can’t specify the variant or pip_url. it’s using the same installed plugin but is applying different config. if you want to use a different version you need to install it and change it’s name to not collide with any other plugin

Thanks @taylor I followed the docs here, are they still correct? I want the different functionality of both variants for now until a few patches land for

meltanolabs

variant. I guess the docs assumes the configs are identical https://docs.meltano.com/guide/plugin-management#multiple-variants

I'm curious, what lock files do you see in

plugins/loaders

?

Running

meltano lock --all

results in

(empiric-elt-py3.10) andycarter@DESKTOP-D6ERPSP:~/vscode_projects/empiric_elt/empiric_meltano/plugins/loaders$ ls
target-jsonl--andyh1203.lock  target-postgres--transferwise.lock

With this meltano.yml:

loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      host: ${SYSTEM_DB_IP}
      port: 5432
      user: ${SYSTEM_DB_USER}
      password: ${SYSTEM_DB_PASS}
      default_target_schema: raw__${MELTANO_EXTRACTOR_NAMESPACE}
      dbname: dwh
      primary_key_required: false
  - name: target-postgres--meltanolabs
    inherit_from: target-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-postgres.git>
    config:
      host: ${SYSTEM_DB_IP}
      port: 5432
      user: ${SYSTEM_DB_USER}
      password: ${SYSTEM_DB_PASS}
      default_target_schema: raw__${MELTANO_EXTRACTOR_NAMESPACE}
      database: dwh

hmm so you're not even getting the lock file for the mlabs variant 🤔

Nope. And logs from a

meltano install --clean

(empiric-elt-py3.10) andycarter@DESKTOP-D6ERPSP:~/vscode_projects/empiric_elt/empiric_meltano/plugins/loaders$ meltano install --clean
Installing 9 plugins...
Skipped installing loader 'target-postgres--meltanolabs'...
Installing extractor 'tap-google-analytics'...
Installing extractor 'tap-mailchimp'...
Installing extractor 'tap-instagram'...
Installing extractor 'tap-facebook-pages'...
Installing extractor 'tap-freshdesk'...
Installing extractor 'tap-spreadsheets-anywhere'...
Installing loader 'target-jsonl'...
Installing loader 'target-postgres'...
Installed loader 'target-jsonl'
Installed loader 'target-postgres'
Installed extractor 'tap-instagram'
Installed extractor 'tap-freshdesk'
Installed extractor 'tap-mailchimp'
Installed extractor 'tap-spreadsheets-anywhere'
Installed extractor 'tap-google-analytics'
Installed extractor 'tap-facebook-pages'
Installed 8/9 plugins
Skipped installing 1/9 plugins

Hi 👋 I noticed the same problem on my local installation. Any thoughts on how to fix this?

@florian_ernst I ended not using the two variants together unfortunately. Hoping with a few more bug fixes I can move off pipelinewise postgres version

Erf... Thanks for the answer 😞 won’t use both variants then

@edgar_ramirez_mondragon can you take a look at this when you get a chance? Seems like we dropped off the thread but there may be a bug in here somewhere

Actually, it seems my issue arise from Metlano’s variant which is not compatible with Python 3.10 (it uses an outdated

botocore

version, which tries to import

from collections import Mapping

when the import moved to

collections.abc

). I also did not manage to install it on 3.9 (Macbook M1) because it throws an error related to the

cryptography

module. In the end, I’m simply using transferwise’s variant.

In this issue there's a workaround for having multiple variants of the same plugin in a project: https://github.com/meltano/meltano/issues/6731. Haven't tested it myself but I think it'd work.

Actually, it seems my issue arise from Metlano’s variant which is not compatible with Python 3.10 (it uses an outdated

botocore

version, which tries to import

from collections import Mapping

when the import moved to

collections.abc

).

Hi, which version of the tap are you using? MeltanoLabs/target-postgres is tested in Python 3.7-3.11.

Apologies, I’m not using

target-postgres

but

target-snowflake

. I’m using meltano and the target’s latest versions. The issue comes from

snowflake.connector

(which in turn calls an outdated botocore).