Hey all. New to Meltano here. I'm writing a tap and want use incremental replication for one of the streams. The endpoint returns paginated results with a

next

property pointing to the next page cursor. Since there's a lot of records returned from this endpoint, I'm trying to limit the amount of requests needed to read all records updated since the last run. The endpoint does not have an

after

param that I can leverage, but I know from the documentation that it sorts records in descending order of

updatedAt

value. So my solution is to keep requesting the next page until I've reached a record where the

updatedAt

timestamp is smaller than the starting replication timestamp. The issue I'm having is the context is not passed to the paginator class anywhere. So I can't do that comparison and exit the pagination loop. What would be the best way to achieve this?

https://github.com/MeltanoLabs/tap-messagebird/blob/main/tap_messagebird/streams.py#L75 check out this implementation

@olivier_caron we have https://github.com/meltano/sdk/issues/1520. Do give it a 👍 if that’d help your use case.

@visch exactly what i was looking for! And @edgar_ramirez_mondragon will do! Thank you both