So I have a problem: I need to replicate table from Postgres A to another Postgres B. But there’s a catch: source table does not have primary keys defined (they exist in data, but without constraints) and data in source table is mutating, so it’s not append-only. I’m using pipelinewise variants of tap and target. With naive incremental approach I get duplicate rows because target does not know which keys to use for deduplication. I tried setting

table-key-properties

in metadata, but pipelinewise-tap-postgres ignores this setting (I confirmed it by looking at code). Questions: 1. Is there a way to force

key_properties

for SCHEMA message? 2. Is there an alternative tap-postgres that behaves better? Maybe based on Meltano SDK?

It may be possible to inject key properties between the tap and target using steam maps.

pipelinewise-tap-postgres

Can you try overriding view-key-properties instead of table-key-properties?

I imagined that problem is here: https://github.com/transferwise/pipelinewise-tap-postgres/blob/master/tap_postgres/discovery_utils.py#L149

Okay, it sounds like the table already has a primary key but you want to use a different key instead - which is not being picked up by the tap, which still prefers its own detected primary key. Can I ask a bit more about the use case? (Just curious.)

@aaronsteers in my case source table has no primary keys defined, so I get empty list for

key_properties

I have no control over source database, so I cannot fix the issue at the source (2) looks nice as a solution, will try thanks!