I am using the `wise` and `meltanolabs` variants of target p Meltano #troubleshooting

I am using the `wise` and `meltanolabs` variants o...

Andy Carter

05/12/2025, 8:07 AM

I am using the

wise

and

meltanolabs

variants of target-postgres in my project, in combination with different taps. I am orchestrating meltano through dagster. I just noticed an occasion where I had an in-progress run with the

wise

variant, and a new run using

meltanolabs

wanted to start.

Copy code

Block run completed.           block_type=ExtractLoadBlocks err=RunnerError("Cannot start plugin target-postgres-meltanolabs: Executable 'target-postgres' could not be found. Loader 'target-postgres-meltanolabs' may not have been installed yet using `meltano install loader target-postgres-meltanolabs`, or the executable name may be incorrect.") exit_codes={} set_number=0 success=False

Is this expected behaviour that only one variant can run concurrently?

Edgar Ramírez (Arch.dev)

05/12/2025, 2:58 PM

That's an interesting setup and it's the first time I see this type of bad behavior. So you did not trigger the

meltanolabs

variant job, but Meltano wanted to invoke that target?

👀 1

Andy Carter

05/13/2025, 7:22 AM

Sorry, to be clear. job_1: tap-a -> target-postgres-wise Job 2: tap-b -> target-postgres-meltanolabs Steps: •

meltano run job_1

(long running) • whilst job above is still running,

meltano run job_2

I want to migrate to labs version eventually, but have a combination of jobs with some taps to wise and some to labs.

Andy Carter

05/13/2025, 7:23 AM

loaders

section:

Copy code

loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
    config:
      destination_path: output/${MELTANO_EXTRACTOR_NAMESPACE}
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      host: ${TARGET_POSTGRES_HOST}
      port: ${TARGET_POSTGRES_PORT}
      user: ${TARGET_POSTGRES_USER}
      password: ${TARGET_POSTGRES_PASSWORD}
      default_target_schema: raw__${MELTANO_EXTRACTOR_NAMESPACE}
      dbname: ${TARGET_POSTGRES_DBNAME}
      database: ${TARGET_POSTGRES_DBNAME}
      primary_key_required: false
  - name: target-postgres-small-batch
    inherit_from: target-postgres
    config:
      batch_size_rows: 10000
  - name: target-postgres-flattening
    inherit_from: target-postgres
    config:
      data_flattening_max_level: 3
      add_record_metadata: true
  - name: target-postgres-meltanolabs
    inherit_from: target-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-postgres.git>
    config:
      database: ${TARGET_POSTGRES_DBNAME}
      dbname: ${TARGET_POSTGRES_DBNAME}
      host: ${TARGET_POSTGRES_HOST}
      user: ${TARGET_POSTGRES_USER}
      password: ${TARGET_POSTGRES_PASSWORD}
      default_target_schema: raw__${MELTANO_EXTRACTOR_NAMESPACE}

Andy Carter

05/13/2025, 1:40 PM

Ah, I think I have a related issue too. I have been seeing an error like

KeyError: 'formatters'

from my target on some

wise

runs. I think this is where I have a

wise

variant job running, and a

labs

run tries to start whilst it is running. The

labs

run fails to start, but might be modifying the loader files in the process of failing. This leaves the python files for the target in an unworkable state, so when the original

wise

job is ready to drop into the target, the target can't be invoked,

Andy Carter

05/13/2025, 1:41 PM

Example logs of the

labs

job that fails to start whilst a

wise

job is running:

Copy code

2025-05-13 13:22:51 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - 2025-05-13T13:22:51.161171Z [info     ] Installing loader 'target-postgres-meltanolabs'
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - 2025-05-13T13:23:05.080007Z [info     ] Logged pip install output to /project/.meltano/logs/pip/loaders/target-postgres/pip.log
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - 2025-05-13T13:23:05.080270Z [error    ] Loader 'target-postgres-meltanolabs' could not be installed: Failed to install plugin 'target-postgres'.
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - 2025-05-13T13:23:05.080380Z [info     ] Collecting git+<https://github.com/MeltanoLabs/target-postgres.git>
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi -   Cloning <https://github.com/MeltanoLabs/target-postgres.git> to /tmp/pip-req-build-0816xnv_
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi -   Running command git clone --filter=blob:none --quiet <https://github.com/MeltanoLabs/target-postgres.git> /tmp/pip-req-build-0816xnv_
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi -   error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi -   fatal: expected 'packfile'
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi -   error: subprocess-exited-with-error
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi -   × git clone --filter=blob:none --quiet <https://github.com/MeltanoLabs/target-postgres.git> /tmp/pip-req-build-0816xnv_ did not run successfully.
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi -   │ exit code: 128
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi -   ╰─> See above for output.
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi -   note: This error originates from a subprocess, and is likely not a problem with pip.
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - error: subprocess-exited-with-error
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - × git clone --filter=blob:none --quiet <https://github.com/MeltanoLabs/target-postgres.git> /tmp/pip-req-build-0816xnv_ did not run successfully.
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - │ exit code: 128
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - ╰─> See above for output.
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - note: This error originates from a subprocess, and is likely not a problem with pip.
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - 2025-05-13T13:23:05.080630Z [warning  ] Installed 0/2 plugins
2025-05-13 13:23:05 +0000 - dagster - INFO - yardi_assets - c334ee3a-c267-41a5-b616-8761717597cc - yardi - 2025-05-13T13:23:05.080746Z [warning  ] Skipped installing 1/2 plugins

Andy Carter

05/13/2025, 1:42 PM

I'm pretty sure that the variants are competing to write/run files to the same

target-postgres

directory when each starts up.

Andy Carter

06/25/2025, 8:33 AM

Hi @Edgar Ramírez (Arch.dev) just wanted to get your view on this to close out an issue. If I have two variants of target-postgres, can I run meltano using them both simultaneously with an orchestrator? The behaviour I'm seeing suggests that the variants are both attempting to install to the same location, so if I start a run with tap-a to

run tap-a target-postgres-wise

variant, then kick off

run tap-b target-postgres-meltanolabs

, the original

tap-a

run fails when trying to load to postgres. It's like the

wise

python code is gone from the plugin folder, and has been replaced with the

meltanolabs

plugin code. Or should I be scheduling to avoid conflict? The end state is to get everything to

meltanolabs

target in a couple months.

Edgar Ramírez (Arch.dev)

06/26/2025, 10:49 PM

The safest way is probably to add them as "custom" plugins:

Copy code

meltano add --from-ref <https://raw.githubusercontent.com/meltano/hub/b09e3352bb19abf6f1ee8f72eaea8b806af22087/_data/meltano/loaders/target-postgres/meltano.yml> loader target-postgres

then modifying the name and variant to avoid conflicts.

Andy Carter

06/27/2025, 8:11 AM

Thanks, appreciate the response.

Open in Slack

Previous Next