Hi all, I'm trying to use two different variants from tap-postgres in my project but a strange error came up. I'm currently using transferwise variant for incremental and log-based pipelines. When I added a new extractor with the meltanolabs variant, the log-based transferwise pipelines kept working, but my two existing incremental pipelines threw this error

database=config["database"], KeyError: 'database'

. meltanolabs variant requires the

database

setting, but these extractors are based on transferwise, which accepts

dbname

setting. Do you have any thoughts? Thanks!

Hi 👋! What does you

meltano.yml

look like?

Hi! I will post the content from three files inside the extract folder. My

meltano.yml

looks like this:

...
include_paths:
- ./environments/*.meltano.yml
- ./extract/*.meltano.yml
- ./load/*.meltano.yml

log-based-extractor.meltano.yml

(still working)

plugins:
  extractors:
  - name: log-based-extractor
    inherit_from: tap-postgres
    variant: transferwise
    pip_url: git+<https://github.com/josescuderoh/pipelinewise-tap-postgres.git>
    load_schema: my_log_based_schema
    metadata:
      '*':
        replication-method: LOG_BASED

incremental-extractor.meltano.yml

(failed after I added the meltanolabs extractor)

plugins:
  extractors:
  - name: incremental-extractor
    inherit_from: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    load_schema: my_incremental_schema
    metadata:
      '*':
        replication-method: INCREMENTAL
        replication-key: id
      public-alerts:
        replication-method: INCREMENTAL
        replication-key: updated_at
      public-api_keys:
        replication-method: INCREMENTAL
        replication-key: updated_at
...

log-based-extractor-meltanolabs.meltano.yml

(I didn't even test it because I had to rollback the changes)

plugins:
  extractors:
  - name: log-based-extractor-meltanolabs
    inherit_from: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
    load_schema: my_log_based_schema_meltanolabs
    metadata:
      '*':
        replication-method: LOG_BASED

Can you try removing the

inherit_from

entry and reinstalling?

If I just remove and run

meltano install

I receive

Extractor 'postgres-labs' is not known to Meltano

. It just works without

inherit_from

if I use

tap-postgres

as the name, but this way I can't define more than one tap with different variants. Just as reference, I tried to follow these instructions: https://docs.meltano.com/guide/plugin-management/#multiple-variants

Oh right, there's a workaround in https://github.com/meltano/meltano/issues/6731#issuecomment-1239944284 that might be worth trying. I'll give it a try later to catch any missing steps.

I haven't tried this approach yet. I will test it today and keep you posted on the results. Thank you!

Can you override the `executable`:

plugins:
  extractors:
  - name: pipelinewise-tap-postgres
    executable: tap-postgres

?

Yes, it worked now. I can execute both extractors and apparently they don't overwrite each other as before. I will configure some ELTs based on this approach and see if everything works fine. Thank you!

Thanks for bearing with me! I've updated the comment in the issue to include the

executable

step.

Thank you!