👋 I'm trying to figure out if I can use meltano/singer taps for my project. I can't seem to find an answer in the docs. I'm trying to replicate large sql tables with a UUID primary key and some late arriving data. In other words, I don't think I can manage state with an incrementing primary key, and using an

updated_at

field would cause it to miss the data that arrives late. Is there a way to manage this?

Hi @sean868! If the tables are too large for constant full refreshes or backfills, sounds like logical replication might be what you need. Most databases should support it (https://hub.meltano.com/extractors/postgres#default-replication-method, https://docs.meltano.com/guide/integration#log-based-incremental-replication)

Thanks. I'm unfortunately using an old and obscure version of SQL, so I think I'm out of luck. (firebirdsql)

I see... So does late arriving data in your case means data may arrive in the wrong order (e.g. 2022-02-18 records arriving after some 2022-02-22 records) or just that there's a general delay. In the latter case, Singer state might still work for you.

There's a general delay. So records from yesterday might not show up until 3 days from now, but today's data will arrive tonight. They get inserted into chronological order when the records finally do arrive