Hi, I'm working on a tap using a restStream. I am confused about how the state is managed. The sdk docs just mention

The SDK was designed to automatically handle the *state* implementations so that developers don't have to.

I don't get what is meant by that. How can we define which state must be kept, and/or what state is kept by default? I would like to manually update the state, as I need to do something similar as described here. Do I need to set

replication_key

and/or

replication_key_value

for this? What I hope to achieve is to update a timestamp in the state everytime I have processed one of the 'bins' in a timerange, as described in the linked slack post. I have overridden

get_next_page_token

to calculate the required parameters, but I don't know how I can update the state now after a batch/bin is finished.

Hi, @mathieu_hinderyckx. Thanks for your question. The built-in handling only requires you to set

replication_key

to the name of your datetime field. Optionally, you can also set

is_sorted=True

in cases where you know and trust 100% that your data will come pre-sorted.

The data should be in pre-sorted order yes. How I need to request the data: do a request with a certain daterange parameter, and within that daterange bin I need to do multiple 'inner' requests since the response is paginated. After all the pages are done I need to move to the next daterange and work through all the pages again. A single value for replication_key / incremental key (so in this case that would be the start of the daterange I suppose) is thus not enough to know where I am in the stream: I need to also know within the daterange on which page I am. The state should thus be a combination of (daterange_start, page).

The state should thus be a combination of (daterange_start, page).

Won't the page be volatile, and likely change on next run though if you start from a different start time/datestamp?