Argo Workflow dynamic fanout 😎 The first task generates a list of all our google analytics tap names (we have 49 UA properties, each with an inheriting tap), and then creates a downstream task to run that meltano command (

meltano run tap-google-analytics--$propertyname target-redshift

). And then when all meltano commands have run, it kicks off a

meltano invoke dbt-redshift build --select models/staging/google_analytics

job to update the dbt models

is it incremental? How do you store state?

in the Argo Workflows definition, there are three tasks: The first task writes a JSON array to stdout, of the form

[{"tap":"tap-google-analytics-1"},{"tap":"tap-google-analytics-2"}]

The second task is parameterized, and generates a task that runs

meltano run $tap target-redshift

for each

tap

value in the output of the first task.