Best practice question from a non-data-engineer: it seems like the majority use-case for ELT is loading data into a warehouse for analysis. Could it also be used to sync data across external services, for example to send data from one SaaS product to another, or is it recommended to implement that a different way? For example, if a user opts-out of email marketing on one service, opting them out of others

@julian_knight That may be a better question for the Singer Slack workspace 🙂 My take is that Singer taps are (obviously) highly data source specific, outputting a known list of streams and properties (~ entities and attributes), while most Singer targets are entirely data source agnostic, and can be fed any arbitrary list of streams and properties that are then translated into tables and columns with the same names, that the target doesn't otherwise need to understand. In your use case, you'd want a data source-specific target, since it would need to know how to interpret specific streams and properties in order to map them to the correct changes on the destination side, and you'd lose the "any tap works with any target" feature of Singer, since your target would really only work with the specific tap(s) it was designed for. There's no reason that wouldn't work though, and I don't think you'd be doing anything particularly egregious, so while there may be other solutions specifically designed for this purpose (just like Singer was specifically designed for extracting and loading stream/property aka entity/attribute aka table/column-style data), there's an obvious advantage in using a tool that's already part of your stack anyway. But if you're going to be doing a lot of this API-to-API type of work, you're probably better off looking for a dedicated tool. Note that I wouldn't mind Meltano developing in the direction of supporting that kind of data transfer natively, but we'd probably want to adopt something other than Singer for that use case.

Thanks @douwe_maan. What about triggering side-effects after a pipeline runs instead, i.e. keep the target data agnostic but fire off a webhook or function call or something to some custom code to do the import?

@julian_knight Then you're getting into the orchestrator's territory and out of Meltano's purview: you'd define a pipeline (or DAG, what have you) with a

meltano elt

step, followed by a step (or operator in Airflow parlance) in which you can run arbitrary code or commands

cool, I think I like that strategy better. I'm going to do a bit more research and see if we can orchestrate stuff like that with airflow. We've got 2 or 3 processes like this now, and I can see them growing in the future, and each has been implemented a different way, so having a consistent way to do stuff like this would make maintaining them easier. Thanks @douwe_maan!

Good luck!

@julian_knight for what you are talking about, Prefect could be a great solution. For example, one of our needs is to trigger a workflow to run ad hoc, rather than on a schedule

@adam_roderick Prefect looks neat. Do you run the meltano dag in prefect along with your custom tasks, and if so, did you do it by hacking on

orchestration/dags/meltano.py

?

@julian_knight I don't think Adam uses

meltano schedule

, so he wouldn't have based his Prefect Flow logic on that Airflow DAG generator, but it shouldn't be too hard to rewrite it to use Prefect Flows and Tasks instead of Airflow DAGs and Operators, since ultimately all it does is iterate over the result of

meltano schedule list --format=json

, and create a new

meltano elt <args>

DAG for each of them

If I understand you right, we took a different approach. Our Prefect DAG runs via Prefect, and it calls

meltano elt ...

as Douwe stated.

ah yeah that makes more sense. thanks y'all!