pierre_de_poulpiquet
03/23/2021, 8:07 PMRESTStream
.
The closest thing that I saw was: https://gitlab.com/meltano/singer-sdk/-/blob/development/singer_sdk/samples/sample_tap_gitlab/gitlab_rest_streams.py#L160 where 2 streams are used and a sort of state (is it the same that the tap state?) is used to pass data from the 2 streams.
In Hubspot, there are thousands or hundred of thousands Contacts Ids to be passed between the 2 streams. Would the recommended solution be the same as the one implemented into the Gitlab tap?
b. Some streams in Hubspot are nested into a single API / Endpoint. Ex. Deal Pipeline and Deal Pipeline Stage are returned through 2 nested lists into a single endpoint.
How would it be mapped with the SDK? Is it possible to create a Stream that consume the results of the API calls of another Stream without doing API calls?
c. What is the concurrency model of the SDK between Streams? Are all streams synced in sequence, or is there some sort of concurrency?
My experience shown that Hubspot paginated APIs have high latency so the ātotal sync timeā can be quite long: weāre spending a lot of time waiting for the next page (and when people have 100k records, that a lot of pagesā¦).
Syncing different paginated streams (ex. Contacts and Deals) at the same time is saving time in this case.
How would the SDK have helped? š