Hi team :wave: I'm evaluating using meltano for so...
# getting-started
r
Hi team đź‘‹ I'm evaluating using meltano for some upcoming projects, and I'm trying to wrap my head around a lot of new stuff - I come from a more old-school luigi ETL background, so apologies for inevitably stupid questions! Supposing I had some kind of job where I needed to make API requests based on row(s) of a db table, parse the response, then store the output. How might this fit into meltano? Am I strictly limited to transforms I can write in SQL? Can I pipe input to taps to control what they output? Or something else?
j
That sounds a little more like you need an orchestration tool like Airflow/Dagster/Prefect to run your Meltano code rather than something that Meltano can handle by itself. I have a similar situation where I need to ingest many different databases on different MySQL instances that are all defined in a db table. I use Dagster as my orchestrator, but you can use any of the ones I mentioned above.
Or another option that may work within Meltano itself (not sure since I don’t have enough details on your use case) is to design a custom tap using Meltano’s SDK using the parent-child streams feature. This allows you to run a stream based on the output of a separate stream and process both the parent and child stream results using your target plugin https://sdk.meltano.com/en/latest/parent_streams.html