One other question regarding the SDK. Are there any examples of handling data that requires another request to get entries? For example, my use case is calling an endpoint to get all

Companies

so I can then retrieve all

Projects

in those companies to populate the

ProjectsStream

In your example, are

Companies

and

Projects

both potential data streams? We are working on a spec for ‘hierarchical’ streams, which would allow parents streams to instantiate child streams. Even if the parent stream is not selected, the parent could iterate ‘on mute’, send its record data to the child stream, and the child stream can use the parent’s data in their own http endpoints.

Yes they are both streams. That is exactly the behavior I am going for

Okay, great! This is a long thread (sorry) but this would be the ticket to follow: add support for hierarchical streams (#97) · Issues · meltano / Singer SDK · GitLab

@aaronsteers So it seems for now this is not possible with the Singer SDK, right?

The way you would have to do it today would be to first initialize

Stream.partitions

on the

Projects

stream with a list dictionaries describing all

Companies

. After defining the class’s partitions, the context of “which Company is the parent of this Project” would get answered in the

partition

argument that is send automatically to most relevant methods. Everything is designed to “just work” after you define what those partitions should be defined as.

Cool, I’ll try that out!

Awesome. And let me know how it goes. 🙂

@aaronsteers Worked like a charm! Wow, I built an entire tap in 2 hours haha this SDK is pretty awesome already

WOW - that’s fantastic! I’m thrilled to hear it.