I think what I'm about to describe is a pattern that the SDK doesn't support, so I'm posting it here for some initial discussion before opening an issue. When creating a REST based tap, an ideal pattern would be to have parameters with a start and end date and then paginate through the results. Unfortunately my experience is that for some APIs, large amounts of data can either make the response times unreasonably slow and/or generate 5xx-type errors. A solution is to "batch" the requests, so that instead of (eg)

2018-01-01

to today's date, you initially request something like

2018-01-01

to

2018-02-01

, paginating through the results and then incrementing your "batch" to the next month etc. As far as I know, there isn't support for this pattern in the SDK right now, and so unless you're prepared to rewire things considerably you're pushed to batch requests through your orchestration layer. I would like to propose an additional layer above (I think)

_sync_records()

that allows for this batching. I'm open to contributing this myself but I'd need some guidance on how best to implement it (assuming y'all think adding this sort of abstraction is a good idea). Also need a better name than 'batching' since that has its own meaning inside the SDK!

I could be completely misunderstanding, but this is the kind of thing that sometimes gets handled in overriding some combination of

get_next_page_token

(to calculate the next start date for example),

get_records

(to figure out how to chunk the "batches"), and ultimately

get_url_params

(to make the next request based on the token or whatever), right? so essentially anything where the backfill has to be iterated through in like a "do x time period per run" kinda way? <-- mostly just checking I understand this right as I've done this a few times in some not-so-nice ways so 👀

If that's the canonical recommendation I'm happy to fly with it, I just thought it might be nice to abstract away the batching given how frequently APIs choke on gracefully paginating large responses.

oh I'm not sure it is by any means 😅 just wanted to chuck it out there to check it was the same thing in the hope that there may/ could be a nicer way!

get_next_page_token

may not always read as the place to look when requesting next interval 🤷

Hey @joshuadevlin! FWIW you may be able to accomplish this using a paginator class. A custom paginator could hold state for both the page and date range by overriding get_next and has_more.

I've found that thus far (Maybe with my lucky selection of taps) that adding good retry logic (I'm willing to wait 10-45 min + ) and having good state management with dates (like https://github.com/AutoIDM/tap-googleads/blob/main/tap_googleads/streams.py ) solves the issue for me. (start date / end date stuff here https://github.com/AutoIDM/tap-googleads/blob/main/tap_googleads/streams.py#L168 ) But having some kind of "binning" methodology could certainly work, it'd be like a layer with pagination like @gary_james mentioned. As @edgar_ramirez_mondragon mentions the paginator class is perfect (I made the google tap just before this was a thing, but indeed uses one) I could have went with your approach for this tap , but instead went with https://github.com/AutoIDM/tap-indeed/blob/main/tap_indeedsponsoredjobs/streams.py#L72 (start date stuff here https://github.com/AutoIDM/tap-indeed/blob/main/tap_indeedsponsoredjobs/streams.py#L82 ) as they support massive per page sizes 🤷 and the response times aren't too nuts Sharing both of these mainly as they may be helpful as they both have start / end date stuff with state. Hope it might help!

This is useful — I hadn't seen the paginator classes at all (I'd been handling pagination manually in the way that Gary outlined above). Let me have a play and see where I get to!

This might help https://github.com/anelendata/tap-rest-api

Thanks Dylan but I’d prefer to use the Meltano SDK