https://linen.dev logo
Join Slack
Powered by
Hey everyone! I was wondering if anyone has encou...
# troubleshooting
g
Hey everyone! I was wondering if anyone has encountered this error before?
Copy code
meltano | Running transformation...
dbt | Running with dbt=0.16.1
dbt | Checking ../.meltano/transformers/dbt/target/*
dbt | ERROR: not cleaning ../.meltano/transformers/dbt/target/* because it is protected
dbt | Checking dbt_modules/*
dbt | Cleaned dbt_modules/*
dbt | Checking logs/*
dbt | Cleaned logs/*
dbt | Finished cleaning all paths.
dbt | Running with dbt=0.16.1
dbt | Encountered an error:
dbt | Compilation Error
dbt | Env var required but not provided: 'MELTANO_LOAD_SCHEMA'
meltano | Transformation failed (2): Env var required but not provided: 'MELTANO_LOAD_SCHEMA'
meltano | ELT could not be completed: `dbt deps` failed
d
dbt requires any environment variables referenced in dbt yml files to be set. https://docs.getdbt.com/reference/dbt-jinja-functions/env_var
If any environment variables are not set, then dbt will raise a compilation error.
d
@gunnar What tap and target are you using? Can you share the meltano.yml definition?
g
@douwe_maan tap-shopify and target-redshift (datamill-co version of target-redshift)
Copy code
loaders:
  - name: target-redshift
    namespace: target_redshift
    pip_url: target-redshift
    executable: target-redshift
    settings:
    - name: redshift_host
    - name: redshift_port
    - name: redshift_database
    - name: redshift_username
    - name: redshift_password
    - name: redshift_schema
    - name: default_column_length
    - name: target_s3.aws_access_key_id
    - name: target_s3.aws_secret_access_key
    - name: target_s3.bucket
    - name: target_s3.key_prefix
Copy code
- name: tap-shopify
    variant: singer-io
    pip_url: tap-shopify
Copy code
transforms:
  - name: tap-shopify
    variant: meltano
    pip_url: <https://gitlab.com/meltano/dbt-tap-shopify.git@config-version-2>
Copy code
transformers:
  - name: dbt
    pip_url: dbt==0.16.1
files:
  - name: dbt
    pip_url: git+<https://gitlab.com/meltano/files-dbt.git>
d
Does your dbt profiles.yml have a profile for redshift? It seems like the env var for schema is not defined. I think if you change it to “MELTANO_LOAD_REDSHIFT_SCHEMA” should work since redshift_schema is the name of the setting, not just “schema”
g
I did create a profile for it in profiles.yml
Copy code
target_redshift__shopify:
      type: redshift
      threads: 2
      database: xxx
      schema: xxx
      host: xxx
      user: xxx
      password: xxx
      port: xxx
Other than that, it is just the default profiles.yml generated by Meltano/DBT. (I was just quickly testing so I didn't use env variables when initially setting this up) is the MELTANO_LOAD_SCHEMA referring to the target schema that the transform would load into? This is the top of the profiles.yml file:
Copy code
config:
  send_anonymous_usage_stats: False
  use_colors: True
meltano:
  target: "{{ env_var('DBT_TARGET') }}"
  outputs:
However, I only applied these changes (creating a profile for redshift) in profiles.yml I do notice though that there is also a 
'profiles (dbt).yml'
file
d
@gunnar I think the tap-Shopify transform expects MELTANO_LOAD_SCHEMA to be defined, which corresponds to the “schema” setting of the loader, but your target doesn’t have a “schema” setting. I don’t know if the transform would work with redshift since it’s written for postgres
g
@douwe_maan Would this be a possible solution to get around it?
d
@gunnar It’s related, but only if your dbt model uses 
MELTANO_LOAD__TARGET_SCHEMA
, but it seems to use 
MELTANO_LOAD_SCHEMA
directly, which is specific to the “schema” setting. Can you find where it uses that env var? That’s what I’d suggest changing
g
I will definitely look into that, thank you!
I did find this section in the dbt_project.yml, however, looks unrelated to the target/load schema.
Copy code
models:
  my_meltano_project: null
  tap_shopify:
    vars:
      schema: '{{ env_var(''DBT_SOURCE_SCHEMA'') }}'
I am using this Meltano provided transform (tap-shopify): https://gitlab.com/meltano/dbt-tap-shopify/-/tree/master/models The above I assume is related to the schema specification here in the screenshot? The models folder in 
/transform/models
is empty which has me thinking that Meltano generates it on each run?
d
The reference to `MELTANO_LOAD_SCHEMA`appears to be coming from here: https://gitlab.com/meltano/dbt-tap-shopify/-/blob/master/dbt_project.yml#L22 That’s the default value ,that should be overridden by the 
schema: '{{ env_var(''DBT_SOURCE_SCHEMA'') }}'
definition in your 
dbt_project.yml
What version of dbt are you using?
g
The default via the Meltano installation: version 0.16.1
d
g
@douwe_maan Since this is being used in Meltano and installed via the 
meltano add
command, does the upgrade need to be done in any specific way? Like changing the reference in meltano.yml file? Or would simply running 
pip install --upgrade dbt
work?
Copy code
transformers:
  - name: dbt
    pip_url: dbt==0.16.1
  files:
  - name: dbt
    pip_url: git+<https://gitlab.com/meltano/files-dbt.git>
d
@gunnar You can update the 
pip_url
for 
dbt
in 
meltano.yml
, and then run 
meltano install transformer dbt
g
Perfect, thank you!
This succeeded and got me to a point where I believe I can continue troubleshooting alone, thank you! I had one last question though: Does this file live in the Meltano project at all? Or is it generated on the fly? I can't seem to locate it
For context: this was my latest error that I believe I can work on fixing:
Copy code
dbt | Running with dbt=0.19.1
dbt | Checking ../.meltano/transformers/dbt/target/*
dbt | ERROR: not cleaning ../.meltano/transformers/dbt/target/* because it is protected
dbt | Checking dbt_modules/*
dbt | Cleaned dbt_modules/*
dbt | Checking logs/*
dbt | Cleaned logs/*
dbt | Finished cleaning all paths.
dbt | Running with dbt=0.19.1
dbt | Installing <https://gitlab.com/meltano/dbt-tap-shopify.git@config-version-2>
dbt | Installed from revision config-version-2
dbt |
dbt | Installing <https://gitlab.com/meltano/dbt-tap-zendesk.git@config-version-2>
dbt | Installed from revision config-version-2
dbt |
dbt | Running with dbt=0.19.1
dbt | [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
dbt | There are 2 unused configuration paths:
dbt | - models.tap_zendesk.vars
dbt | - models.tap_shopify.vars
dbt |
dbt | Found 26 models, 15 tests, 0 snapshots, 0 analyses, 163 macros, 0 operations, 0 seed files, 0 sources, 0 exposures
dbt | The selector '' does not match any nodes and will be ignored
dbt | The selector 'tap_shopify__products' does not match any nodes and will be ignored
dbt | The selector 'my_meltano_project' does not match any nodes and will be ignored
dbt | WARNING: Nothing to do. Try checking your model configs and model specification args
meltano | Transformation complete!
d
@gunnar That file does not live in your repo, but is automatically installed by 
dbt deps
which looks at the packages in 
transform/packages.yml
. I think it installs it into 
transforms/dbt_modules
That latest error has to do with https://gitlab.com/meltano/meltano/-/issues/2824: because you’re using an inherited 
tap-shopify
, it isn’t matching the 
transform
tap-shopify
when running 
meltano elt
with 
--transform run
g
Ah I see, I checked there and it was not populated, however I think on the latest run it finally generated. Thank you!
d
You can fix that by explicitly setting the dbt 
models
setting to 
$MELTANO_EXTRACT__MODEL_NAME
, and then adding 
model_name: tap_shopify
to your 
tap-shopify--products
definition
And I’m now seeing we have a doc on that here: https://hub.meltano.com/transformers/dbt#no-models-running 😄
g
Amazing! This did help resolve many of the warning and errors:
Copy code
meltano | Extract & load skipped.
meltano | Running transformation...
dbt | Running with dbt=0.19.1
dbt | Checking ../.meltano/transformers/dbt/target/*
dbt | ERROR: not cleaning ../.meltano/transformers/dbt/target/* because it is protected
dbt | Checking dbt_modules/*
dbt | Cleaned dbt_modules/*
dbt | Checking logs/*
dbt | Cleaned logs/*
dbt | Finished cleaning all paths.
dbt | Running with dbt=0.19.1
dbt | Installing <https://gitlab.com/meltano/dbt-tap-shopify.git@config-version-2>
dbt | Installed from revision config-version-2
dbt |
dbt | Installing <https://gitlab.com/meltano/dbt-tap-zendesk.git@config-version-2>
dbt | Installed from revision config-version-2
dbt |
dbt | Running with dbt=0.19.1
dbt | [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
dbt | There are 2 unused configuration paths:
dbt | - models.tap_shopify.vars
dbt | - models.tap_zendesk.vars
dbt |
dbt | Found 26 models, 15 tests, 0 snapshots, 0 analyses, 163 macros, 0 operations, 0 seed files, 0 sources, 0 exposures
dbt | The selector 'tap-shopify' does not match any nodes and will be ignored
dbt | WARNING: Nothing to do. Try checking your model configs and model specification args
meltano | Transformation complete!
I appreciate all the help @douwe_maan, I think these last steps just need configuration/updated references (If I'm not mistaken). I'm new to DBT so I'm just figuring some of this out as I go haha
d
@gunnar Glad I could help!
g
@edward_ryan @monika_rajput @jo_pearson @tom_mcgrail @kai_yokoyama
e
Thanks so much @douwe_maan
Open in Slack
PreviousNext
Powered by