Howdy, has anyone successfully implemented state i...
# singer-tap-developmente
eric_boucher
06/27/2022, 2:55 PMHowdy, has anyone successfully implemented state incrementation for streams that go in DESCENDING order? In other word, I am hoping to remember the state during the first page, and only update it at the end of the pagination. So that next calls can use this state as a “since” parameter
eric_boucher
06/27/2022, 3:58 PMcc @edgar_ramirez_mondragon https://github.com/MeltanoLabs/tap-github/pull/152
Would love your input 🙂
eric_boucher
06/28/2022, 5:14 PM@edgar_ramirez_mondragon let me know if you have any questions. Indeed the goal is to wait until a complete iteration over get_records before updating the state. Since we can’t update the state until we know for sure that we have paginated successfully to the beginning. That’s the main issue with descending streams is that they are not “resumable”
p
philip_johnson
09/07/2022, 3:48 PM@eric_boucher did you ever figure this out? Running into a similar issue where I'm unable to sort by ascending order and so the state is prematurely set on the first page and the 2nd api call returns no new records
e
eric_boucher
09/10/2022, 10:37 AMThe PR above is our latest attempt unfortunately and it did not work completely as expected. I think it goes in the right direction so feel free to have a look though and let me know if you have questions.
You can see how we use
use_fake_since_parameterp
philip_johnson
09/10/2022, 2:37 PMThanks! I did stumble upon this and think Meltano could just be magically handling this, but it's hard to tell: https://sdk.meltano.com/en/latest/implementation/state.html#dealing-with-unsorted-streams
I still can't get my streams to run properly (they just die for no reason) and can't get the help I need so think I'm going to move to a different solution myself
2 Views