Good morning (UK here) What is the recommended way of implementing incremental replication where the tables have a UUID field as

id

? I see that the query is modified with

ORDER BY id

which makes no sense with UUIDs so can I specify a compound field of

created_at

with

updated_at

or even some custom SQL like

COALESCE(updated_at, created_at)

? I'm referring to the

replication-key

property

metadata:
  <table>:
    replication-key: <value>

Edit: The reason for this question is because the application design is so poor that the

updated_at

field is not always set when a row is created. This is why the coalesce statement is used to get the latest version of a row. Hence my question 🙂

How about log-based replication? We would prefer that usually over key based for incremental. That way you leverage the source to determine the change sets.

One way if you are able, define a view on the db that does the coalescing of created & updated, and then select from that view? But guessing if you have this issue on one table, you will have it elsewhere...

Thanks @Sven Balnojan I will be trying that next. I'm working my way through the replication methods to get an understanding of limitations and strengths as I get to know Meltano (one week in so far)

I've seen other questions related to this but cant find them at the moment. You can use stream maps to coalesce 2 fields after theyve been extracted like I do here but that would only really help with deduping and would still require extracting the data from the source each time. I've heard similar requests enough times for it to at least deserve a feature request issue in the SDK

Awesome @pat_nadolny thank you!