Hi. I’m using pipelinewise-tap-postgres against pipelinewise-target-postgres. I have a table that’s missing a primary key, and it’s causing duplicates in my target. I’ve added explicit table-key-properties to the metadata for the stream, but it doesn’t seem to do anything. Does anyone know how to fix this?

I've had similar problems with managing key properties with the pipelinewise postgres variants. I use the meltanolabs

meltano-map-transformer

mapper to deal with the issue.

Interesting. We use that for redacting fields. How can I use it for my current problem?

-

meltano add mapper meltano-map-transformer

add a config like this:

plugins:
  mappers:
  - name: meltano-map-transformer
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    executable: meltano-map-transform
    mappings:
    - name: mapper_for_adding_key_properties
      config:
        stream_maps:
          thenameofthestream:
            __key_properties__:
              - primary_key_column_no_1
              - primary_key_column_no_2

Oh that’s wild.

works reliably for me. (I use it the other way around though, I need to remove key properties from the stream)

Testing it out now.

omg it worked. Can I buy you a coffee?

At the current price of coffee futures, that might be a good investment to make. 😂