I realised a flaw in my incremental setup In the source syst Meltano #troubleshooting

I realised a flaw in my incremental setup. In the ...

mert_bakir

09/16/2023, 12:35 PM

I realised a flaw in my incremental setup. In the source system updated_at field is null when the records first created. This is not a problem for the initial full sync but then I realised

replication_key_value

(the value stored in the meltano state) falls behind. For example in the source system the max value for the updated_at field is in august, so new records does not get inserted. Is there a way to configure this properly, like telling the tap to use

coalesce(updated_at, created_at)

mert_bakir

09/17/2023, 6:11 AM

Rows with NULL values in the Replication Key column will only be replicated during the first extraction of an integration. This means subsequent extractions will not capture rows where the Replication Key is NULL.

https://www.stitchdata.com/docs/replication/replication-keys According to this page replication key should not be null, so I have to handle this in the source system. Is this true, there is no way around?

mert_bakir

09/18/2023, 4:30 PM

@edgar_ramirez_mondragon Sorry for disturbing but can you confirm that I have to make sure replication-key to be not null? Is there another way to handle this with tap configurations?

edgar_ramirez_mondragon

09/18/2023, 10:21 PM

Hi @mert_bakir! If you control the tap, you could override Stream.post_process:

Copy code

def post_process(row, context):
   row["updated_at"] = row.get("updated_at") or row["created_at"]
   return row

as described in https://github.com/meltano/sdk/issues/1198

Open in Slack

Previous Next