trying to run `meltano invoke dbt run` on a docker container Meltano #troubleshooting

trying to run `meltano invoke dbt:run` on a docker...

josh_lloyd

06/21/2021, 4:15 PM

trying to run

meltano invoke dbt:run

on a docker container (through dagster), but I get the following error:

Copy code

dagster.core.definitions.events.Failure: Shell command execution failed with output: Executable 'dbt' could not be found. Transformer 'dbt' may not have been installed yet using `meltano install transformer dbt`, or the executable name may be incorrect.

I ran

RUN meltano install

in the Dockerfile during the docker build step and I can log into the container and see that

.meltano/transformers/dbt/venv/

is populated with all the seemingly relevant packages. The

invoke

command works when I run it from my local terminal. Also, a

meltano elt …

command runs just fine in the dagster docker instance except when I add the

--transform=run

command. It errors with the same error message when it gets to the transform step. Relevant portion of

meltano.yml

file:

Copy code

...
plugins:  
...
  transformers:
  - name: dbt
    pip_url: dbt==0.19.1

douwe_maan

06/21/2021, 4:20 PM

@josh_lloyd Can you confirm there’s an executable at

.meltano/transformers/dbt/bin/dbt

josh_lloyd

06/21/2021, 4:21 PM

yes

Copy code

root@4f85690f794a:/opt/dagster/app/.meltano/transformers/dbt/venv# ls bin/
__pycache__  activate.csh   chardetect  jp.py       keyring  pip3    pybabel  pyrsa-decrypt  pyrsa-keygen    pyrsa-sign    python   slugify               snowflake-dump-ocsp-response        snowflake-export-certs  wheel
activate     activate.fish  dbt         jsonschema  pip      pip3.7  pyjwt    pyrsa-encrypt  pyrsa-priv2pub  pyrsa-verify  python3  snowflake-dump-certs  snowflake-dump-ocsp-response-cache  sqlformat

douwe_maan

06/21/2021, 4:23 PM

@josh_lloyd Now that you’re in the container, can you run

meltano install transformer dbt

again, and see if that fixes it and lets you run

meltano invoke dbt

again?

josh_lloyd

06/21/2021, 4:24 PM

wrong josh 😉, but yeah, give me a sec

douwe_maan

06/21/2021, 4:25 PM

😅 Slack should really rank people in the same thread before randos when I type

@josh

josh_lloyd

06/21/2021, 4:26 PM

🙂 no kidding! didn’t work:

Copy code

root@4f85690f794a:/opt/dagster/app# meltano invoke dbt
Executable 'dbt' could not be found. Transformer 'dbt' may not have been installed yet using `meltano install transformer dbt`, or the executable name may be incorrect.

douwe_maan

06/21/2021, 4:26 PM

Try with

meltano --log-level=debug …

douwe_maan

06/21/2021, 4:26 PM

Let’s see if we can get some more stacktrace info

josh_lloyd

06/21/2021, 4:28 PM

output.txt

douwe_maan

06/21/2021, 4:29 PM

Copy code

FileNotFoundError: [Errno 2] No such file or directory: '/opt/dagster/app/transform': '/opt/dagster/app/transform'

douwe_maan

06/21/2021, 4:30 PM

Does that directory exist?

douwe_maan

06/21/2021, 4:30 PM

Where does your

dbt_project.yml

live?

josh_lloyd

06/21/2021, 4:31 PM

oh, yeah you’re right I forgot to copy that dir in the dockerfile

douwe_maan

06/21/2021, 4:31 PM

😄

douwe_maan

06/21/2021, 4:31 PM

That’ll do it!

josh_lloyd

06/21/2021, 4:31 PM

haha. Love dumb user errors 😄

douwe_maan

06/21/2021, 4:35 PM

Glad we were able to figure it out 🙂 The error message was pretty misleading in this case

josh_lloyd

06/21/2021, 4:40 PM

thanks a ton for the help!!

8 Views

Open in Slack

Previous Next