Hello, I’m pretty new to Singer. Are there any working examples of how to use state.json? I’ve set the replication key to the page value of a query as a test. I capture state from the output and write it to state.json. However, when using

--state state.json

the code seems to always perform a full table replication. I’m unclear if the bookmarking state logic is built in, or if I need to implement it in sync() as I’m seeing with a lot of the existing metlano-optimized taps.

Hi @john_hwang! Are you using the SDK?

Yes I am using the SDK.

are you using

get_starting_replication_key_value

or

get_starting_timestamp

in your code? see https://sdk.meltano.com/en/latest/classes/singer_sdk.Stream.html?highlight=get_starting_replication_key_value#singer_sdk.Stream[…]lication_key_value

I was not, thanks for the advice. As suggested, I used

get_starting_replication_key_value

inside get_url_params, and print the results of the function. I get the expected replication_key_value of 25.

Nice!

It’s a relief to know that my state.json is at least being read in correctly. But isn’t the incremental update process supposed to automatically skip queries with lower replication_key values? Why is it still emitting records for pages 1-24?

I’m assuming I just have to manually implement the incremental replication?

That is correct. It's up to the dev to implement the actual filtering that the source system expects 👍

Also, if it’s relevant, I’m running the tap using poetry rather than using meltano run or meltano elt.

Yup, that should be alright.

Ok thanks for the feedback!

Let me know if you get stuck 🙂