Hi. I’ve got some data enrichment processes that don’t neatly fit into the Meltano ‘T’ / dbt framework - namely, they are written in python. I’d love to use

PL/Python

in a

dbt

transformation, but its not an official PostgreSQL plugin, not supported by AWS RDS, and likely neither functional nor practical within

dbt

. Currently, it runs regularly via a CRON job. The most straightforward solution is to call it via an

Airflow operator

That said, I don’t see best practices documented at Meltano regarding this type of setup. Does anyone here have a similar setup? How do you think about running crucial transformations that don’t neatly fall into the `tap`/ `target`/`dbt` categories in the context of Meltano?

We're collecting issues regarding non-dbt transformations in https://gitlab.com/groups/meltano/-/epics/84, e.g. https://gitlab.com/meltano/meltano/-/issues/2300. That issue covers Python transformations that sit in between the E and L steps, but it sounds like your script would need to be run after EL has completed? We could explore adding native support for those (post-EL transformers other than dbt) in a separate (related) issue, if you'd like to create one 🙂 For the moment, such a setup would require you to move away from having Meltano manage your entire pipeline, to having

meltano elt

just be a step in your manually crafted Airflow DAG, followed by whatever additional steps you'd like. This is the approach also commonly taken by people who already have Airflow and dbt set up, and just want to bring in Meltano for the EL step.

Let me read through these threads. Thanks, Douwe. My work requires public records data to be gathered in from many different cities / states. When we scale up from 1 to many states, my plan is to add 1 meltano instance per state. these instances will be coordinated by a meta-orchestrating Airflow. The current design dilemma is whether to coordinate those post-EL transformers by the state-specific meltano instance, or via the meta-orchestrator. Sounds like the latter is the right solution, for now. I appreciate your help and will open an issue if need be.