Hi all I m trying to do something which doesn t seem natural Meltano #singer-tap-development

Hi all. I''m trying to do something which doesn't ...

laurent

01/19/2023, 12:46 PM

Hi all. I''m trying to do something which doesn't seem "natural" with

tap-github

, and I'm hoping to find some better way to do it: • the stream hierarchy in the tap has

Repositories

at the top, and

dependencies

as a child stream. • I need to fetch the

repositories

stream for each repo that comes out of

dependencies

. right now I'm doing this by processing the output of

dependencies

and feeding it as input to a separate run of the rap Is there some obvious/built-in (in the sdk) way to tell the tap to run a given stream (that is not a child) for each result?

aaronsteers

01/21/2023, 8:43 PM

Hi, @laurent. How about a subclass or near-clone of the

Repositories

stream class, named something like

DependentRepositories

(

'dependent_repositories'

), mapped as a child of the

dependencies

stream with no child streams of its own?

aaronsteers

01/21/2023, 8:52 PM

On the other hand, if want to traverse child streams of those repos, a much more ambitious approach would be to add settings like

recurse_dependent_repos

(bool) and

max_recursion_depth

(int). In theory at least, it may be possible to make "Repositories" stream it's own "grandpa" in the hierarchy, so long as you add some loop detection during discovery and to skip already-seen entities in

get_child_context()

. 🤓

laurent

01/23/2023, 10:12 AM

Hi @aaronsteers thanks for those ideas. I think that first option should be enough for my needs at this point.

2 Views

Open in Slack

Previous Next