Another question, the code for transformations live in dbt repo outside of meltano project right? If that is the case, how do you currently manage deployments and code updates. How does dbt repo code updates get pushed to meltano production? Please feel free to point me to docs

You can choose to have your dbt models live wherever you like, inside the

transform/models

directory or in separate repos or directories referenced from

transform/packages.yml

, which is where Meltano automatically adds any

transform

plugins referenced in

meltano.yml

. From that point on, the models and packages are managed entirely by

dbt

, not Meltano, and dbt will pull by URL automatically when Meltano runs

dbt compile

and

dbt run

as the last step in the ELT pipeline. If I'm understanding how dbt works correctly, that means that you wouldn't need to push dbt model changes to production, because they'll be pulled automatically

yeah let me setup a pipeline and run dbt models on it and see how things go

@nehil_jain in a recent project, we run our dbt transforms outside of meltano, and we use meltano for extract/load. Our orchestrator knows when to call

meltano elt

commands, and when to call

dbt run

commands. It is a great setup and I recommend it if you are trying to work with an existing production setup