Hi everyone, I wanted to thank you for developing the singer-sdk. However, I encountered an issue with incremental replication, I have the following scenario: - I have successfully exported the data from the tap to the target - in the database (postgreSql) the state is saved by setting the value of the replication key correctly I cannot extrapolate the value of the replication key to perform a new incremental elt. Is there a function of the singer-sdk that allows me to retrieve the value of the replication key saved in the database in order to use it in the next call? Thank you😀

Are you calling

meltano elt

with a

--job_id

parameter?

Hi, @rocco_melillo. Thanks for the inquiry. Within the SDK, I think what you are looking for is Stream.get_starting_replication_key_value() and Stream.get_startime_timestamp(), depending on if your key is based on a timestamp (and therefor should also leverage

start_date

) or not. The docs linked above have recently been expanded to better explain this, but probably we should make this more discoverable in other parts of the docs/templates as well.

@edward_smith yes I am using always the same job_id to do incremental 🙂 @aaronsteers resolved using Stream.get_startime_timestamp(),😀 my key is a timestamp Thank You for your replies!

Fantastic! 🎉

Hi @aaronsteers, I am joining the thread to ask you another question about this function. I noticed that it returns None in case the last elt-job failed (state = FAIL). Would it be possible to obtain the value of the replication key of the last job in the Success state in order to continue the incremental import in case of error (eg quota excedeed)? Thank you

Hi, @rocco_melillo. There's actually a bunch of magic inside the spec to make sure the state only passes through the target (thereby becoming available as a bookmark again in the tap). Also, the implementation depends upon whether the stream is sorted by the incremental key or not. If the stream is not defined as sorted, then only fully-complete ("finalized") stream operations are marked resumable. If the stream is sorted, than the target can emit state once it's "caught up" with writing all the rows it's seen so far.