When running `target postgres` is there a common reason for Meltano #troubleshooting

When running `target-postgres` is there a common r...

fraser__dagster_labs_

03/10/2023, 7:37 AM

When running

target-postgres

is there a common reason for encountering this error:

Copy code

psycopg2.errors.UndefinedTable: relation "temp_LNS14000000_073143d1_2387_48f8_b46e_b7790e025d07" does not exist

While I have been ingesting data from some test sources successfully, one of my taps will not load to postgres and keeps throwing this psycopg2.errors.UndefinedTable error.

visch

03/10/2023, 1:16 PM

Hey @fraser__dagster_labs_ ! Are you using the pipelinewise or meltanolabs variant?

visch

03/10/2023, 1:50 PM

Smells like pipelinewise, and since it's a temp table I wonder if it's session bound or something? Like maybe your target ran for a long time and it lost connection so the session doesn't have access to the temp table 🤷 Complete guess

fraser__dagster_labs_

03/10/2023, 3:00 PM

Thanks - I am running the

meltanolabs

variant, and I am simply following the intro tutorial, so the

meltano.yml

reads:

Copy code

- name: target-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-postgres.git>
    config:
      add_record_metadata: true
      database: postgres
      user: meltano
      host: localhost

visch

03/10/2023, 3:02 PM

😓 hmm, that target does work, unless something really crazy happened. You just ran through the tutorial and you're seeing that? I can run through it and see if I can replicate what you're seeing I guess, any changes from the tutorial?

visch

03/10/2023, 3:32 PM

@fraser__dagster_labs_ I just had a second and gave it a shot, I didn't have a problem 😕

Copy code

version: 1
send_anonymous_usage_stats: true
default_environment: dev
project_id: target-postgres
plugins:
  extractors:
  - name: tap-smoke-test
    namespace: tap_smoke_test
    pip_url: git+<https://github.com/meltano/tap-smoke-test.git>
    executable: tap-smoke-test
    config:
      streams:
      - stream_name: animals
        input_filename: <https://gitlab.com/meltano/tap-smoke-test/-/raw/main/demo-data/animals-data.jsonl>
      - stream_name: page_views
        input_filename: <https://gitlab.com/meltano/tap-smoke-test/-/raw/main/demo-data/pageviews-data.jsonl>
      stream_maps:
        animals:
          __key_properties__: [id]
        page_views:
          __key_properties__: [vistor_id]
  - name: tap-github
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-github.git>
    config:
      repositories:
      - sbalnojan/meltano-lightdash
      start_date: '2022-01-01'
    select:
    - commits.url
    - commits.sha
    - commits.commit_timestamp
  loaders:
  - name: target-postgres
    namespace: target_postgres
    pip_url: -e .
    settings:
    - name: sqlalchemy_url
      kind: password
    config:
      host: localhost
      port: 5432
      user: postgres
      password: postgres
      database: postgres
      target_schema: test
      add_record_metadata: true
environments:
- name: dev

Ran

meltano run tap-github target-postgres

fraser__dagster_labs_

03/10/2023, 4:08 PM

I ran the intro tutorial with the tap-github and loaded data to the Postgres instance in Docker just fine. I then switched to a different Tap (tap-bls) The data loads fine to

target-jsonl

but when I try to ingest the same to

target-postgres

(with same settings as the tap-github run) i run into those

psycopg2.errors.UndefinedTable

issues. So yeah, clearly related to the format of the incoming json. I was just curious if this was a known issue with an obvious solution . Thanks for confirming that this all should work… I will keep on investigating on my end

Open in Slack

Previous Next