https://linen.dev logo
Join Slack
Powered by
Is it possible to use stream maps to set the repli...
# plugins-general
s
Is it possible to use stream maps to set the replication key for a stream?
✅ 1
e
Not currently. I'm curious what's the use case. Is it something like overriding a broken tap's replication key?
s
Some taps like the sql server tap are default full table replication unless you specify the replication key in the catalog. I wanted to find a way around that without having to specify the full catalog. Using 
metadata
achieves this:
Copy code
- name: tap-mssql
    variant: wintersrd
    capabilities:
      - state
    metadata:
      <STREAM_NAME>:
        replication-method: INCREMENTAL
        replication-key: <DESIRED_REPLICATION_KEY>
this is the tap: https://github.com/wintersrd/pipelinewise-tap-mssql I didn't want to have to pass in a full catalog into 
properties.json
, 
metadata
is pretty handy way around this! You can also set the primary key(s) in the metadata:
Copy code
metadata:
      <STREAM_NAME>:
        replication-method: INCREMENTAL
        replication-key: <DESIRED_REPLICATION_KEY>
table-key-properties: [<PRIMARY_KEY>]
e
Yeah, 
metadata
is the way to go! I was curious if this was for a tap that doesn't accept catalog overrides. So to confirm, does 
metadata
solve this for you?
s
yes this solves it for me!
👌 1
12 Views
Open in Slack
PreviousNext
Powered by