https://linen.dev logo
Join Slack
Powered by
Hello! I’m upgrading an old (~0.50) configuration ...
# troubleshooting
p
Hello! I’m upgrading an old (~0.50) configuration and I’m hitting a weird env error with 
meltano elt tap-mysql--incremental target-bigquery --transform=run
.
Copy code
2022-01-25T03:36:31.348398Z [info     ] The selection criterion '' does not match any nodes cmd_type=transformer job_id=one name=dbt run_id=31fdd9c5-2235-470c-be38-7574a88c22bb stdio=stderr
2022-01-25T03:36:31.349839Z [info     ] The selection criterion 'tap_mysql__incremental' does not match any nodes cmd_type=transformer job_id=one name=dbt run_id=31fdd9c5-2235-470c-be38-7574a88c22bb stdio=stderr
2022-01-25T03:36:31.351275Z [info     ] The selection criterion 'my_meltano_project' does not match any nodes cmd_type=transformer job_id=one name=dbt run_id=31fdd9c5-2235-470c-be38-7574a88c22bb stdio=stderr
Based on how the command gets generated, looks like 
MELTANO_TRANSFORM__PACKAGE_NAME
is empty. setting that env to the name from dbt_project.yml, I still get these errors:
Copy code
2022-01-25T03:50:30.590396Z [info     ] The selection criterion 'tap_mysql__incremental' does not match any nodes cmd_type=transformer job_id=one name=dbt run_id=16265c3b-0104-42ae-bad7-54767f54717a stdio=stderr
2022-01-25T03:50:30.591884Z [info     ] The selection criterion 'my_meltano_project' does not match any nodes cmd_type=transformer job_id=one name=dbt run_id=16265c3b-0104-42ae-bad7-54767f54717a stdio=stderr
Which suggests to me I’ve just got something misconfigured. This is all coming through from the 
discovery.yml
, and I’m just using the default one that gets put in 
.meltano/cache/
on install. The offending line is:
Copy code
transformers:
- name: dbt
  ...
  - name: models
    value: $MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project
I’m going to try stripping out the dbt config and re-adding using the latest in https://docs.meltano.com/guide/transformation, but that may take a while, and I’m suspicious that I’m just missing something simple — so if something jumps out in the above, please do let me know!
I’m now getting this unhelpful error message:
Copy code
2022-01-25T19:28:52.844109Z [info     ] Running transformation...      job_id=one name=meltano run_id=a2eacb06-9821-45f5-af43-dc21b498f4db
2022-01-25T19:28:52.852178Z [info     ] ELT could not be completed: Cannot start dbt: 'project_dir' cmd_type=elt job_id=one name=meltano run_id=a2eacb06-9821-45f5-af43-dc21b498f4db stdio=stderr
Here we go, with debug logging:
Copy code
Traceback (most recent call last):
  File "/Users/paul/.pyenv/versions/3.7.9/envs/bi-pipeline/lib/python3.7/site-packages/meltano/core/runner/dbt.py", line 39, in invoke
    stderr=asyncio.subprocess.PIPE,
  File "/Users/paul/.pyenv/versions/3.7.9/envs/bi-pipeline/lib/python3.7/site-packages/meltano/core/plugin_invoker.py", line 278, in invoke_async
    async with self._invoke(*args, **kwargs) as (
  File "/Users/paul/.pyenv/versions/3.7.9/envs/bi-pipeline/lib/python3.7/site-packages/async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "/Users/paul/.pyenv/versions/3.7.9/envs/bi-pipeline/lib/python3.7/site-packages/meltano/core/plugin_invoker.py", line 263, in _invoke
    popen_options = {**self.Popen_options(), **kwargs}
  File "/Users/paul/.pyenv/versions/3.7.9/envs/bi-pipeline/lib/python3.7/site-packages/meltano/core/plugin/dbt/base.py", line 20, in Popen_options
    return {**super().Popen_options(), "cwd": self.plugin_config["project_dir"]}
KeyError: 'project_dir'
It seems somehow the 
invoke
call isn’t getting any config passed in.
Poking around in the internals, it seems that nothing is setting 
project_dir
in the settings, and this key is not getting set in the 
invoker.project_dir
. @douwe_maan any pointers on how you’d expect these keys to get populated? The docs say this should be defaulted, but 
project_dir
isn’t making it into the DBT invoker.
d
@paul_tiplady Can you share your 
meltano.yml
definition for dbt?
p
I’ve tried a few iterations. Originally it was simply:
Copy code
plugins:
  ...
  transformers:
  - name: dbt
    pip_url: dbt==0.21.1
I just tried this one, and now it’s complaining about “command clean could not be found”
Copy code
transformers:
  - name: dbt
    namespace: bi_pipeline
    pip_url: dbt==0.21.1
    config:
      project_dir: /Users/paul/qwil/bi-pipieline/transform/
Which suggests I’m overriding the discovery.yml spec for DBT somehow?
(The second one is what I get if I do 
meltano config dbt set project_dir $(pwd)/transform/
)
d
The 
namespace
is throwing Meltano off and making it think you’ve got a complete definition instead of inheriting from the discoverable one
Do you see the same error without the namespace?
If you remove that, manually setting 
project_dir
may not be necessary anymore and that error may go away
p
K, thanks — that’s fixed the proximal issue.
Open in Slack
PreviousNext
Powered by