I'm working with a stream that returns a list of records, which is being used as parent stream for a child that requests a record at a time. The delta in run time between incremental and full sync is ~6 hours so I'm trying to make this incremental if I can.

The parent stream returns records in descending order of created date (the records are immutable so we don't need to worry about updates). There's no ability to filter the records.

My current approach is: • On the first "page" save the first (most recent) timestamp to a tap attribute • At the end of the run write that timestamp to the state dictionary (manually) • On incremental runs, check whether the last (oldest) timestamp in each page is before the timestamp from the state dictionary and if so, don't fetch any more records.

This feels a bit hacky (particularly writing the state manually), so I'm wondering if there's an simpler approach that I'm missing.

This might not directly pertain to your use case but perhaps there could be helpful info in there if you haven't read it already.

We should add this use case to the docs I just linked to. It's definitely a case that comes up periodically and would be good to have a documented best practice approach.