https://linen.dev logo
Join Slack
Powered by
So overwriting/setting the key_properties attribut...
# troubleshooting
s
So overwriting/setting the key_properties attribute for a stream should be possible via the key_properties field inside the metadata, right? @edgar_ramirez_mondragon @pat_nadolny @aaronsteers @visch. For some reason, no matter how I spell "key_properties"/"key-properties"/"table-key-properties", when I invoke my tap the schema message always has an empty key_properties field. (https://hub.meltano.com/singer/spec/). Does anyone have a working example for me maybe? (I got a postgres table in my case)
v
Copy code
version: 1
send_anonymous_usage_stats: true
project_id: "tap-postgres"
plugins:
  extractors:
  - name: "tap-postgres"
    namespace: "tap_postgres"
    pip_url: -e .
    capabilities:
    - state
    - catalog
    - discover
    config:
      sqlalchemy_url: "<postgresql://postgres:postgres@localhost:5432/postgres>"
    settings:
    - name: sqlalchemy_url
      kind: password
    select:
    - "public-test_users.*"
    metadata:
      "public-test_users":
        key_properties: ["id"]
Works for me! Tested by removing the primary key and going back and forth. I will say it did take me a bit to get the syntax here right, should probably add this as an example in the docs under metadata?
s
Thanks! I got to say I do find https://hub.meltano.com/singer/spec/#metadata a bit confusing ("key_properties" is no metadata in that list), so definitely add an example somewhere please.
Although my example still doesn't work, did I miss something essential here?
Copy code
- name: tap-postgres
    variant: transferwise
    capabilities:
      - state
      - discover
      - catalog
    pip_url: pipelinewise-tap-postgres
    config:
      host: 172.17.0.1
      port: 5432
      user: admin
      password: password
      dbname: postgres
    select:
      - "public-orders.*"
    metadata:
      "public-orders":
        replication-method: FULL_TABLE
        #replication-key: id
        key_properties: ["cid"] # different id...
a
s
Hm @aaronsteers that's even more confusing 😄 because it doesn't for for my example, and @visch has a working example that uses key_properties and not table-key-properties.
v
Probably something to do with the different variants. You're using pipelinewise mine is using the meltanolabs version. Not sure which is/isn't right 😮 I"ll try it with pipelinewise maybe 
table-key-properties
works for that one?
a
Let's iterate from the stack overflow answer, and change as needed. That's the only authority I know of as of now - and I logged that a while back exactly for slack threads like this one. 🙂 Since this is a passthrough to the tap, I agree it's possible that it's possible different variants may expect different inputs (although hopefully that's not the case), and it's also possible that more than one spelling will work. If all else fails, we can go look at the source code. 😅
I'll also log an issue to put this formally into the docs.
s
I'm headed into a week of vacation now, but I'll also try after I'm back 🙂
a
@Sven Balnojan - can you confirm if 
table-key-properties
works for your transferwise tap-postgres example? If not, can you log to the issue above?
v
fwiw
Copy code
- name: "pipelinewise-tap-postgres"
    namespace: "tap_postgres"
    pip_url: "pipelinewise-tap-postgres"
    executable: "tap-postgres"
    capabilities:
    - state
    - catalog
    - discover
    config:
      host: "localhost"
      port: 5432
      user: "postgres"
      password: "postgres"
      dbname: "postgres"
    select:
    - "public-test_users.*"
    metadata:
      "public-test_users":
        table-key-properties: ["name"]
        key_properties: ["name"]
        replication-method: "FULL_TABLE"
Doesn't work for me either @Sven Balnojan to help with the issue
a
Maybe a bug in the tap then?
v
Yes I'm pretty certain
7 Views
Open in Slack
PreviousNext
Powered by