I’m a little stuck with a weird rest api pattern: ...
# singer-tap-developmenth
Henning Holgersen
05/02/2023, 6:08 PMI’m a little stuck with a weird rest api pattern: The endpoint uses a HATEOAS (“next” link in json response) paginator, but: The initial query (when the run starts) is POST, but when following the “next” link, the request has to be GET. Has anyone implemented something like that?
Henning Holgersen
05/02/2023, 6:15 PMWait a sec, the problem might be something else entirely…
Henning Holgersen
05/02/2023, 6:46 PMOK I’m back to confusion. I have implemented a hateoas class (which I have done in a few other taps as well), but these other taps have only used GET.
Now, what I see, is that it loops on the first page. The debug logs aren’t all that informative, so I will need to dissect it more to get info on what gets called. Any experience with HATEOAS and POST are appreciated.
Henning Holgersen
05/02/2023, 7:03 PMOK I just confirmed that the problem is that it picks up the first “next” link correctly, but it doesn’t update. And it doesn’t catch that it gets caught in a loop.
e
edgar_ramirez_mondragon
05/02/2023, 7:14 PMbut it doesn’t updateHi Henning! Can you elaborate on this? Is the main issue here that subsequent requests need to be
GETPOSTh
Henning Holgersen
05/02/2023, 7:22 PMI thought the POST->GET thing was the problem, and it still might be, but I’m not sure anymore.
Basically: Each request returns 50 records, and when I run the tap the first 100 records are unique, the rest are just a repeat of records 51-100. I log the “next” URL and verified that this URL is the same the whole way.
It’s getting late over here, but the two theories I have:
1. A basic mistake on my part made it loop
2. The combination of POST+GET somehow takes the SDK for a spin which also bypasses the loop-detection.
I’ll pick it up again tomorrow when my brain is more functional.
Henning Holgersen
05/02/2023, 7:23 PMAnd yes, I verified manually with postman that this isn’t a bug in the API. It is not the API that loops.
Henning Holgersen
05/03/2023, 11:19 AMNow that I’m awake again, I realized: The pagination I set up, that I thought picked up the entire URL, only tacked the query params from the returned URL on to the end of the original URL. For some weird reason, this worked somewhat, a strange buggy undocumented behaviour that accounts for yesterday’s mystery.
So the question becomes much simpler: How do I make sure I use the whole URL in the “next” field without any strange attempts at parsing?