https://linen.dev logo
Join Slack
Powered by
Has anyone ever worked with variables in DBT? I'm ...
# plugins-dbt
s
Has anyone ever worked with variables in DBT? I'm trying to set a variable, but it doesn't seem to register
Getting error:
Copy code
Could not render {{ var('TARGET_BIGQUERY_PROJECT_ID_PROD') }}: Required var 'TARGET_BIGQUERY_PROJECT_ID_PROD' not found in config:
   Vars supplied to <Configuration> = {}
t
where are you setting it? https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/dbt_project.yml#L64-91 is an example on the gitlab data team project
s
I'm setting it in dbt_project.yml
Copy code
vars:
  TARGET_BIGQUERY_PROJECT_ID_DEV: potloc-x
  TARGET_BIGQUERY_PROJECT_ID_STAGING: potloc-xxx
  TARGET_BIGQUERY_PROJECT_ID_PROD: potloc-xxxxxx

target-path: ../.meltano/transformers/dbt/target
log-path: logs
packages-install-path: dbt_packages
clean-targets:
- ../.meltano/transformers/dbt/target
- dbt_packages
- logs
models:
  meltano_transformer:
    +database: |
      {%- if  target.project == "potloc-dev" -%} {{ var('TARGET_BIGQUERY_PROJECT_ID_DEV') }}
      {%- elif target.name == "staging"  -%} {{ var('TARGET_BIGQUERY_PROJECT_ID_STAGING') }}
      {%- elif target.name == "prod"  -%} {{ var('TARGET_BIGQUERY_PROJECT_ID_STAGING') }}
      {%- else -%} invalid_database
      {%- endif -%}
t
it could be a quoting thing? I’m not sure about the multi-line syntax
I do see that you have 
{%- elif target.name == "prod"  -%} {{ var('TARGET_BIGQUERY_PROJECT_ID_STAGING') }}
which seems like it should point to the prod variable
On the multi-line issue - I wonder if tryign the folded style with 
>
would work better? ¯\_(ツ)_/¯
I use https://learnxinyminutes.com/docs/yaml/ as a reference for all things yaml
s
Basically my goal was to replace env_var with vars since I don't want to manage too many environment variables in our production environment
t
Understood. Where are you referencing 
TARGET_BIGQUERY_PROJECT_ID_PROD
?
s
As a variable, only there As an env_var, in all of my src_<source_name>.yml files
SInce I'm doing inter-project dbt, I need to specify my source database directly in the file
t
What command / file is generating the `
Copy code
Could not render {{ var('TARGET_BIGQUERY_PROJECT_ID_PROD') }}: Required var 'TARGET_BIGQUERY_PROJECT_ID_PROD' not found in config:
   Vars supplied to <Configuration> = {}
error?
want to hop on a quick zoom to see if we can debug this?
s
dbt_project.yml
Copy code
Compilation Error
  Could not render {%- if  target.project == "potloc-dev" -%} {{ var('TARGET_BIGQUERY_PROJECT_ID_DEV') }}
  {%- elif target.name == "staging"  -%} {{ var('TARGET_BIGQUERY_PROJECT_ID_STAGING') }}
  {%- elif target.name == "prod"  -%} {{ var('TARGET_BIGQUERY_PROJECT_ID_STAGING') }}
  {%- else -%} invalid_database
  {%- endif -%}
  : Required var 'TARGET_BIGQUERY_PROJECT_ID_STAGING' not found in config:
  Vars supplied to <Configuration> = {}
stephane@stephane-XPS-15-9510:~/Document
t
oh I wonder if variables aren’t availabe in that context yet
s
I would love to but am currently in a meeting 😞
Would it be possible that meltano doesn't take into account variables?
t
Meltano is running the full dbt package so there really shouldn’t be a meltano bug there. are you using 
meltano run
or 
meltano invoke
?
s
meltano invoke dbt:run --select my_model
t
hmm, yeah that just runs the dbt executable. I’d say try adjusting the yaml block type but then also look in the dbt logs to see if there’s more info. you can also run with 
--log-level=debug
on the meltano side to see if more info spits out
s
Sadly not; I'll continue investigating I guess 😉
SOLVED IT: https://github.com/dbt-labs/dbt-core/issues/2904 need to set a default value
Thanks for all your help @taylor
t
whew
10 Views
Open in Slack
PreviousNext
Powered by