Howdy, has anyone successfully implemented state i...
# singer-tap-development
e
Howdy, has anyone successfully implemented state incrementation for streams that go in DESCENDING order? In other word, I am hoping to remember the state during the first page, and only update it at the end of the pagination. So that next calls can use this state as a “since” parameter
cc @edgar_ramirez_mondragon https://github.com/MeltanoLabs/tap-github/pull/152 Would love your input 🙂
@edgar_ramirez_mondragon let me know if you have any questions. Indeed the goal is to wait until a complete iteration over get_records before updating the state. Since we can’t update the state until we know for sure that we have paginated successfully to the beginning. That’s the main issue with descending streams is that they are not “resumable”
p
@eric_boucher did you ever figure this out? Running into a similar issue where I'm unable to sort by ascending order and so the state is prematurely set on the first page and the 2nd api call returns no new records
e
The PR above is our latest attempt unfortunately and it did not work completely as expected. I think it goes in the right direction so feel free to have a look though and let me know if you have questions. You can see how we use
use_fake_since_parameter
when streams are being traversed in descending order
p
Thanks! I did stumble upon this and think Meltano could just be magically handling this, but it's hard to tell: https://sdk.meltano.com/en/latest/implementation/state.html#dealing-with-unsorted-streams I still can't get my streams to run properly (they just die for no reason) and can't get the help I need so think I'm going to move to a different solution myself