I have a REST API that has a 'download' report pat...
# singer-tap-developmentj
joshuadevlin
02/04/2022, 2:59 PMI have a REST API that has a 'download' report pattern:
• Hit the API to trigger a report
• Poll an endpoint to see when the report is ready
• Hit another endpoint to get a download URL
• Download a CSV/JSON from that URL
I'm pretty sure the answer is no, but is there any way to use the SDK to handle this pattern? This is for an existing tap I've built using the SDK (where all of the other endpoints follow a standard REST pattern, eg the data is returned directly with pagination etc)
v
visch
02/04/2022, 3:13 PMThe simple approach would be to parent child this
1. Parent stream (hits api to trigger report)
2. Child Stream (Polls the endpoint to get the download url) Give this some "reasonable" timeout period maybe 5 minutes so the job doesn't run forever, download the file.
A more complicated one would be to store the report url in state or something so you'd survive tap failures, but I think the simple approach would do what you're after
visch
02/04/2022, 3:15 PMhttps://sdk.meltano.com/en/latest/code_samples.html#custom-response-validation could be used to do the waiting on your child stream 🤷
Should get you close atleast
visch
02/04/2022, 3:16 PMCould make a case that this could be one stream, and just do all the querying in get_records.
I think I like that better than parent/child streams here as there's no parent data you care about at all. One stream, do the waiting per record in get_records
p
pat_nadolny
02/04/2022, 3:41 PMYeah +1 on just doing it in
get_recordsj
joshuadevlin
02/04/2022, 5:46 PMAwesome, thanks yall!
joshuadevlin
03/03/2022, 7:57 PMJust wanted to circle back and let everyone know that this was really helpful!!
v
visch
03/03/2022, 7:58 PM😄 nice, what approach did you end up going with!
j
joshuadevlin
03/03/2022, 8:43 PMI pretty much jammed all the logic into
get_records()r
Reuben (Matatika)
03/09/2022, 5:43 PMRan into this same issue for `tap-auth0`:
https://github.com/Matatika/tap-auth0/blob/11ed91798248acddde35d9f5d559969d1449d67f/tap_auth0/streams.py#L17
I ended up overriding
RESTStream::_request