https://linen.dev logo
#plugins-general
Title
# plugins-general
d

david_gauthier-dussureault

09/30/2022, 6:40 PM
I'm using the tap-mysql transferwise variant to ingest data from some MySQL views. No primary keys are define in these views, there are no reliable timestamps for incremental replication and I also don't have access to the binary logs for log-based incremental replication. I'm left with full table replication, that's fine. But since there are no primary key defined, when the data is loaded in the target (target-snowflake transferwise variant if that matter), each time the pipeline is triggered I get a new full copy of the views because it only use the
COPY INTO
statement in Snowflake. Is there a way to pass a composite primary key to the tap so that the operation in Snowflake is a
MERGE
on the composite key instead of a full insert? I search to figure if there was a way to add a composite primary key in the tap and I found this thread. Based on this, it seems like I can configure the
table-key-properties
of the entities, but this haven't worked for me so far, it still
COPY INTO
instead of
MERGE
. Here is the config I have:
Copy code
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
    metadata:
      my_entity:
        table-key-properties:
        - Country
        - DeviceType
        - EndDate
        - Organization
        - SerialNumber
        - StartDate