I did a bit of searching, but didn't find anything relevant quickly. Can we configure multiple DBT transforms that are kicked off in sequence, or in parallel if desired, in a single Meltano ELT run? It looks like we can configure multiple transforms in the same meltano.yml file. Sounds like the transform run will be the same name as the extrator (tap-whatever) pulled from package dbt-tap-whatever. Conceptually the goal is to separate the transform of an input from a tap into canonical form from the transform of canonical data for analysis. For instance, we may, and do, have multiple inputs for a category of data, but the format from the various endpoints is all different. In keeping with the ELT philosophy, we would extract and load as close to the original source format as possible, then transform. We would define a canonical form for the data, and the initial transform would be from the disparate original forms into the canonical. Then, we would also need to do analytics on the canonical data. The analysis performed may differ over time based on business needs, but the transformation of the original data to canonical form will likely change much less often, if at all. That is one of the many reasons for keeping these transforms separate. How are others handling this?

We’re in the middle of a large refactor to enable this sort of workflow muuuuch better. Currently Meltano pipelines aren’t composable and the T is too tightly coupled to the EL. In November we should have fully composable pipelines so you can do any set of transformations with dbt you need to in any order and the flow will be much more intuitive. https://gitlab.com/groups/meltano/-/epics/116 is the epic where we’re tracking

I do this right now by taking the "canonical" form, I would call it my stage model (in my case I actually point to a model in another table for System versioning) so I do something like 1. Meltano runs on tap-abc to target-mssql 2. Then I run

dbt run -m +versioningtable.tap-abc

Bam done! I am almost certain you can do this today with the --transform option it would take a few slight tweaks. I don't use the transform option in DBT for this right now because I"m doing this on windows and having to jump through a few hoops but If I were on linux I'd find a way to make it all one command

meltano elt tap-abc target-mssql --transform run --job_id=a

Before the end of November you should be able to do

meltano run tap-abc target-mssql dbt:commands

all within a specific environment context 🎉

very nice, and I have a chance at this on Windows (🤞 )