https://linen.dev logo
Join Slack
Powered by
Using target-postgres-meltanolabs - I have this er...
# troubleshooting
a
Using target-postgres-meltanolabs - I have this error when deploying on Azure in a container.
Copy code
name=target_postgres level=CRITICAL message=Primary key is set to mandatory but not defined in the [PaymentPlan] stream
However, I can't replicate the same error when running locally with same tap and target, so quite difficult to diagnose and fix. Any idea why this might not be happening locally? The stated stream does not have a primary key (although it probably should have), but the sync completes fine without raising the error when running locally.
✅ 1
e
That error doesn't exist in the MeltanoLabs variant, but it does in the Pipelinewise variant:
a
Oh... I think that explains a lot. I have both variants in my project. I have been getting some inconsistent behaviour between local and cloud - column sorting etc. Am I doing this the right way if I want both variants? I am trying to progressively move over to meltanolabs. https://docs.meltano.com/guide/plugin-management/#multiple-variants
Copy code
- name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      host: ${TARGET_POSTGRES_HOST}
      port: ${TARGET_POSTGRES_PORT}
      user: ${TARGET_POSTGRES_USER}
      password: ${TARGET_POSTGRES_PASSWORD}
      default_target_schema: raw__${MELTANO_EXTRACTOR_NAMESPACE}
      dbname: ${TARGET_POSTGRES_DBNAME}
      primary_key_required: false
  - name: target-postgres-meltanolabs
    inherit_from: target-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-postgres.git>
    config:
      database: ${TARGET_POSTGRES_DBNAME}
      dbname: ${TARGET_POSTGRES_DBNAME}
      host: ${TARGET_POSTGRES_HOST}
      user: ${TARGET_POSTGRES_USER}
      password: ${TARGET_POSTGRES_PASSWORD}
      default_target_schema: raw__${MELTANO_EXTRACTOR_NAMESPACE}
I have a task here that is being run by Dagster in the form 
meltano run somedata
Copy code
- name: somedata
  tasks: 
  - tap-somedata target-postgres-meltanolabs
But if as you suggest, the wise variant is being executed, then it looks like this is translating into 
meltano run tap-somedata target-postgres
instead. Bit more digging for me to do.
In the logs I can see lines like 
job_name=dev:tap-somedata-to-target-postgres-meltanolabs
which looks like the right meltano command is being executed. But elsewhere in the logs section for the same run I have an error like this 
File "/project/.meltano/loaders/target-postgres/venv/lib/python3.10/site-packages/target_postgres/__init__.py", line 209
Which seems to suggest the python code in target-postgres is being executed, not the ML variant.
@Edgar Ramírez (Arch.dev) When you use a variant - does it actually reinstall over the parent venv? If I was alternating runs between wise and ML variants, would it be flipping between installing each variant to the same 
.meltano/loaders/target-postgres/venv/
folder? I can see a ``.meltano/loaders/target-postgres-meltanolabs` folder but it is empty
Also I am setting 
ENV MELTANO_PROJECT_READONLY 1
in my Dockerfile, which might mean the variant can't be installed when required in container. Does this conflict with inherited plugins?
I am also running meltano 3.5.0 locally but 3.4.2 in cloud, which I think makes a difference, aligning both to 3.6.0
OK I think I have this sorted now. I am seeing logs like this in my cloud deployment. 
2025-02-12T10:21:02.728254Z [info     ] Installing loader 'target-postgres-meltanolabs'
Not sure if that's down the meltano version bump to 3.6.0, or removing 
ENV MELTANO_PROJECT_READONLY 1
from Dockerfile - either way, happy to have it fixed. Thanks for the pointer Edgar
e
3.5.3 had a few bug fixes for auto-installed inherited plugins, so that might've been the issue. Glad you got it sorted out!
a
Thanks. Is there an conflict with setting 
MELTANO_PROJECT_READONLY
in docker and using 2 variants of the same plugin? If meltano needs to install the variants to the same folder , can this happen with 
MELTANO_PROJECT_READONLY
set?
e
I don't think there'd be any conflict
👍 1
10 Views
Open in Slack
PreviousNext
Powered by