is there a way to have SDK-based taps be resumable for streams that are reverse sorted? i’m trying to update this stripe tap to be resumable and the Stripe API’s

list

endpoints return data reverse-sorted by my replication key with no option to customize that behavior as far as i can tell

Do you know if the API supports getting data in batches, perhaps a month at a time so each month (although sorted descending in terms of individual records) could still form an ascending bookmarkable unit?

the replication key is a timestamp and i can pass a start and end value. how would i do that? would i be writing logic to chunk up the date range into batches and updating state myself or is there something in the SDK to support something like this?

I don't know of any samples doing this today, but in theory something like this could work: 1. Use a custom dict as your next_page_token and use that token to track and inner and outer loop. The outer loop is weeks, days, or months and the inner lip is the built-in pagination token. 2. At each step in the outer loop (for instance, after completing each week of the outer loop), it should be safe to save a bookmark at that point. 3. At least in theory, the finalize_state() method can be called after each step in the outer loop, marking that position as resumable to the downstream.

That sounds like it should work! I'll let you know when I give it a shot. Thanks AJ

Great! And in the process, if you have a tiny bit of time to contribute back to our docs, would be really helpful to add an example of "inner+outer loop pagination" to the code samples page on sdk.meltano.com. Similar topics comes around periodically regarding advanced pagination, and it would be super helpful to have some demo which new devs could use as a reference.

absolutely! probably going to try the approach you suggested next week but i’d be happy to contribute an example

Hey Prratek, you are using target-bigquery, right?

yes - are you thinking it might be related to this? https://meltano.slack.com/archives/C01V8L0NZC1/p1645481978427519

Yea, I worked with Ruslan on

merge_state_messages

since the default target behavior did some merging behavior with state. Wasn't sure if that was impacting what you are seeing

yeah let me set

merge_state_messages

and see if that helps

yea, set it to False

@prratek_ramchandani - re:

however, the next run starts right back at the beginning if that stream doesn't run to completion

For referse-sorted streams, they wouldn't be resumable until after a given stream completes its iteration. What you could do to mitigate would be to wrap in a timeblock that can still be ascending and then finalize the state market after the stream is caught up at least to that checkpoint.