What are best practices for dev-staging-prod environments using dbt and meltano (I'm using bigquery, but this is a general question)? Do you use a project for raw data and a project for results? In which project do you test new taps?

I highly recommend https://courses.getdbt.com/courses/fundamentals

Thanks!

But maybe that's not the question you have here so I'm just guessing

I'll definitely look into it; I need all the dbt training I can get. My question is more aimed at how you setup your production and dev environments in your target, and how do you implement new meltano taps / dbt models

My first guess is that the practices for dev-test-prod and related topics will vary a bit depending on the scale and velocity of your specific environment (and the data in question)

Yes this is what I'm realizing; I was working blind trying to organize my data, but I think I'm going to follow the dbt suggested approach. Update @visch indeed, this tutorial is blowing my mind 🤯

My preferred approach: Iterate quickly! i.e. Do what works. Learn as you go. Fix problems as they arise. e.g. Regarding dbt project structuring, I was aware of the excellent guidelines on dbt from the blogs, but started without reading all of them to show value of dbt. As my dbt projects grew, I quickly started adopting a lot of the recommendations from the blogs that Derek linked.

That's probably the best approach, especially with the ever evolving landscape 😉 thanks @christoph!

@Stéphane Burwash if its helpful for you, I actually created a of our isolated development environments last week! It allows us to have environment and developer level isolation. We use snowflake but most of the configuration is just toggling schema/database names based on meltano environment configs so it should be pretty generically useful

Awesome thank you so much! I'll add it to my to-watch list 😉 I wanted to commend you on your videos btw; I very much enjoyed the dbt-jaffle shop one

Thanks! I'm glad theyre helpful