https://linen.dev logo
Join Slack
Powered by
Can I have multiple dbt projects in one meltano pr...
# plugins-dbt
m
Can I have multiple dbt projects in one meltano project? I see that the dbt-ext utility supports a configurable project directory https://github.com/meltano/dbt-ext/tree/main , I am thinking I could use plugin inheritance to set up dbt-redshift-1 and dbt-redshift-2, will that work?
p
@Matt Menzenski I dont see a reason why this wouldnt work. Have you tried? I was just tinkering with it and was able to inherit but I'm getting 
Could not find adapter type snowflake!
which is strange. The base plugin works and the inherited one executed except it cant find the adapter 🤔 .
a
@Matt Menzenski - In theory, you should be able to do this. The plugin inheritance should allow you to override the dbt project dir.
m
I haven’t tried this yet, and it’ll probably be a bit before we actually try this (if we do, not sure yet).
but we seem to be thinking about having separate Redshift databases for our “intermediate” and “mart” layers and so I’m wondering if this is something we should be looking at
since I believe dbt can only write to one database?
a
dbt should be able to read and write to/from multiple databases
You might be able to handle the intermediate and mart layers without issue.
m
my preference would definitely be to have a single dbt project spanning everything, that would be better for lineage, documentation, etc
a
đź‘Ť
In the past, I've used overridden the 
generate_database_name
macro that is described there, so that in CI/CD, everything shares a single DB (with schema name prefixes providing additional namespaces), and then in prod, I can have an intermediate database (we called it "PREP_DB") and multiple mart DBs ("SALES_DB", "FINANCE_DB", etc.) - and all managed by the same project.
m
that’s great to know, that’s the sort of thing we’d like to have as well
a
Even without the 
generate_database_name
override, you can use the 
database:
property to publish artifacts to different target databases. (I think you can do this based on subfolders within your project, which makes management easier.)
m
we are doing one database per source system (our platform, google analytics, zendesk, etc), all owned by the meltano user, and then an intermediate db and a mart db
4 Views
Open in Slack
PreviousNext
Powered by