Meltano

What happens with tap_postgres if you use key-based incremental loads with a key that is not unique, but is monotonically incrementing?

eg. There is an annoying many-to-many table I need to load incrementally but has only (message_id, recipient_id) as the columns.  Rows are written to this table for each message and are never modified

woudl it work if I increment on message_id, even tho it’s not unique you reckon&gt;?

I think this will work as long as only the latest message_id can have new recipients. If that's not true, you will miss data. You will need to dedupe in dbt as well.

Matt beat me, wrote this out so here you go :smile:

What about the case where tap_postgres runs and you have
```messageid: 1
recipient_id: 12

messageid: 1
recipient_id: 13```
Postgres pulls that data, all is well

Next run it doesn't touch messageid: 1 again but another row was added

```messageid: 1
recipient_id: 12

messageid: 1
recipient_id: 13

message_id: 1
recipient_id: 14```

If there is a `created` timestamp or similar, you may be able to sync with it as your replication key.

yeah, since this is a record of “smss sent” I htink the rows are basically immutable

however to my dismay after trying this I found that the message_id and recipient_id are UUIDs!  So logical replication here we come