https://linen.dev logo
Join Slack
Powered by
I'm not sure if this has already been discussed - ...
# announcements
a
I'm not sure if this has already been discussed - my Slack search is failing me. But I had a question about how Meltano runs dbt models. It seems that each when a pipeline schedule has 
transform: run
, it will by default do a 
dbt run
for every dbt model within the 
transform/models
directory. Is this correct? This seems wasteful - when multiple pipelines are run they're currently building the same dbt models multiple times. I'd like to only build certain models for a given schedule. For example, if I have pipeline called 
foo
, I could set it to only build models under 
transform/models/my_meltano_projct/bar
(equivalent to doing 
dbt run --models bar
). Is there any way to set this up?
d
It seems that each when a pipeline schedule has 
transform: run
, it will by default do a 
dbt run
 for every dbt model within the 
transform/models
 directory. Is this correct?
Yep, the default value of 
dbt
's 
models
setting is 
$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project
, and 
my_meltano_project
refers to the local dbt project with models in 
transform/models
.
For example, if I have pipeline called 
foo
, I could set it to only build models under 
transform/models/my_meltano_projct/bar
 (equivalent to doing 
dbt run --models bar
). Is there any way to set this up?
Definitely! You can change the 
models
setting for all dbt runs using 
meltano config dbt set models <your --models argument>
, or for a specific 
meltano elt
pipeline by setting the 
DBT_MODELS
env var:
Copy code
meltano config dbt set models bar

export DBT_MODELS=bar
And as you an see in the default value, you can reference properties of the extractor as well using env vars: http://meltano.com/docs/configuration.html#expansion-in-setting-values
I'll eventually get to documenting this explicitly: https://gitlab.com/meltano/meltano/-/issues/2258
a
Great, thanks @douwe_maan! I'll give this a try
one more question - is it possible to specify a similar setting for reports/dashboards? If I do a 
meltano add dashboard dashboard-shopify
, for example, I think the dashboard definition will be saved in 
/analyze/dashboards
at the top level. It'd be good to store it in 
/analyze/dashboards/dashboard-shopify
d
If I'm not mistaken, the UI currently only looks for dashboards and reports in the root of the 
analyze/{dashboards,reports}
directories, so we'd need to change that first
It currently expects that all dashboard/report names are unique in the project, without taking into account the source they're for, or any directory they're in
a
Ah yes you're right, thanks Douwe. This isn't that important right now but I just thought I'd check
going back to the 
DBT_MODELS
env var - to use this, would I need to remove my current default schedule(s) and replace them with a custom 
DBT_MODELS=bar meltano elt tap-foo target-bar
command for this to work?
d
No, you can use the 
env
dictionary under 
schedules
: https://meltano.com/docs/integration.html#pipeline-specific-configuration
Or you could add a custom extra to your extractors like 
dbt_models: bar
, which you can then reference from the value of 
models
in 
meltano.yml
using 
$MELTANO_EXTRACT__DBT_MODELS
That way you won't have to explicitly set it on the pipeline, but you can do it dynamically through a property on the extractor
a
ah awesome, thanks! Both sound like good options which would work for me
7 Views
Open in Slack
PreviousNext
Powered by