Hi all We re switching from `tap postgres` `variant transfer Meltano #troubleshooting

Hi all! We're switching from `tap-postgres` `varia...

TomasB

06/07/2023, 8:21 PM

Hi all! We're switching from

tap-postgres

variant: transferwise

tap-postgres

variant: meltanolabs

. The reason we're doing it is that

transferwise

variant ignores the selection of columns that we would like to extract. Using

meltanolabs

variant there're two issues popping up: 1. Getting

Invalid date: '2022-01-17T00:00:00+00:00' Field: <column-name>; Value: 2022-01-17T00:00:00+00:00

when the column on postgres side is a

date

type. 2. Getting CRITICAL No primary keys specified from the tap and Incremental option selected
when we're extracting data from a view in postgres even though we define it in the metadata to look for

Copy code

view-key-properties:
- id

Does anyone have encounter the same issue? Any workarounds?

visch

06/09/2023, 2:43 AM

Can you share your meltano.yml ?

TomasB

06/09/2023, 2:47 AM

What are you looking specifically? Have multiple files. This is the meltano.yml one

Copy code

version: 1
default_environment: dev
project_id: 7b40891b-0b68-40c9-9de9-d76c72d67e1a
send_anonymous_usage_stats: false
include_paths:
- ./environments/*.meltano.yml
- ./extract/*.meltano.yml
- ./load/*.meltano.yml
- ./orchestrate/*.meltano.yml
- ./transform/*.meltano.yml

TomasB

06/09/2023, 2:48 AM

This is the configuration for the tap-postgres that is being used for the views

Copy code

- name: tap-postgres--views
  inherit_from: tap-postgres
    config:
      default_replication_method: INCREMENTAL
      metadata:
        public-*:
          replication-key: updated_date
          view-key-properties:
          - id

TomasB

09/13/2023, 7:37 PM

Not sure if you had any suggestions on this?

visch

09/13/2023, 8:14 PM

What have you tried in 3 months? I'd recommend reposting and showing the errors? 1. There was a date fix at some point, maybe you have the same issue? not clear, if not after a clean install of tap postgres (

meltano install --clean

) , please create a way to reproduce your issue and make an issue on the repo 2. You're using a config variable for metadata overrides, that shouldn't happen in config that should happen in metadata which also makes me think we don't have the real meltano.yml files here maybe a copy paste mess up? I think the main key's in metadata should be

table_key_properties

TomasB

09/13/2023, 8:18 PM

I was using

FULL_TABLE

truncate

replication as a work around but the data got too big that it's taking long time to sync. Looks like

view-key-properties

don't work for views, I just used

key-properties

and it appears to work. Looked at the meltano sdk code and I didn't find

view-key-properties

being used anywhere

Open in Slack

Previous Next