https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
r

Reuben (Matatika)

06/23/2022, 12:24 PM
Is there a way with the SDK to define a kind of stream relationship that can generate records from chaining multiple requests? The API I'm dealing with has an endpoint that returns a list of entities, each with an
id
. There is a separate endpoint to get a set of entity features for a single entity by
id
, or multiple (paged) entities by query parameter
ids
as a comma-separated list. I want to merge the entity and entity features schemas, which means that I need to make an additional request to get a page of entity features for each page of entities. A parent-child stream doesn't feel optimal here given that it would be setup to fire off an entity features request for each entity, which forces a back-off implementation to deal with rate-limiting caused by improper use of the API. I want to leverage existing SDK features in order keep functionality like request authentication and error handling, which becomes much more of a pain to set up when manually invoking
requests
. I ended up instantiating a "secondary" stream inside my "primary" stream and calling
request_records
, but was wondering if there was a better way to do this. Hope I explained that well enough! 😅 https://github.com/Matatika/tap-spotify/blob/4d064c83dc862c9531c4a3ef6046352906f4d22b/tap_spotify/streams.py#L41-L76