Hey team, is there an out-of-the-box way of buildi...
# singer-tap-development
n
Hey team, is there an out-of-the-box way of building a child stream which only runs against a subset of the records returned by the parent stream? The specific example I'm dealing with has a parent stream with a "type" attribute on each record. I'd like to create a child stream which only generates a API calls for a specific "type" of parent record.
a
Hi, @niall_woodward. If you include
type
in the context sent the the child stream (probably via
get_child_context()
, yes, you can just use that as a signal to short-circuit.
I did this in a github implementation, where the issues had a
comments
counter at the issue level. So the child stream knew if count of comments was zero at the parent-level, we'd skip any calls to try pulling comments from that stream. (Saved a lot on runtime and rate limits.)
n
Thanks @aaronsteers - just so I fully understand what you mean by short-circuit, is there a provided function to do that? Any doc links would be 👌
a
I don't think this use case is specifically documented, but you can see a sample here emitting a custom child context and here leveraging that to skip unnecessary calls.
Does that help?
n
Exactly what I was after - thank you