Hello, I was trying to make a docker image to run our ingestion with meltano and transformations with dbt. My problem is that meltano doesnt seem to be able to run dbt when inside a docker container (i have no problem doing this on my pc). If I try to start any dbt operation from the meltano command it always fails saying it couldn't find the 'dbt' executable. However I can see the dbt installation under the .meltano/transformers directory and I can run dbt if I call its script directly, only meltano isn't able to. I tried this with multiple base images, first trying python:3.10-slim, then python:3.10, 3.11, 3.12, ubuntu:latest, meltano/meltano:latest but it always consistently fails with the same error. I'm always using meltano 3.9.1 This is the configuration i have for dbt inside my meltano.yml:

plugins:
  ...
  transformers:
  - name: dbt
    variant: dbt-labs
    pip_url: dbt-core==1.10.11 dbt-bigquery==1.10.2
    config:
      project_dir: $MELTANO_PROJECT_ROOT/../transformation/dbt
      profiles_dir: $MELTANO_PROJECT_ROOT/../transformation/dbt/profiles

jobs:
- name: ingestion
  tasks:
  - tap-mssql target-bigquery
  - tap-rest-api-msdk target-bigquery
- name: transformation
  tasks:
  - dbt:build
- name: source_freshness
  tasks:
  - dbt:freshness
- name: ingest_and_transform
  tasks:
  - tap-mssql target-bigquery
  - tap-rest-api-msdk target-bigquery
  - dbt:build

with this i can easily run dbt using meltano on my pc, for example:

(.venv) user@personal-computer:/meltano-project$ meltano invoke dbt --version
Environment 'dev' is active
Core:
  - installed: 1.10.11
  - latest:    1.10.11 - Up to date!

Plugins:
  - bigquery: 1.10.2 - Up to date!

but the same test fails when run inside a docker container:

root@28ec0fcfb4f8:/project# meltano --log_level=debug invoke dbt --version
Meltano 3.9.1, Python 3.11.13, Linux (x86_64)
Environment 'dev' is active
Creating DB engine for project at '/project' with DB URI 'sqlite:////project/.meltano/meltano.db'
Found plugin parent
Skipped installing transformer 'dbt'
Skipped installing 1/1 plugins
Variable '$MELTANO_LOAD__TARGET_SCHEMA' is not set in the provided env dictionary.
Variable '$MELTANO_TRANSFORM__PACKAGE_NAME' is not set in the provided env dictionary.
Variable '$MELTANO_EXTRACTOR_NAMESPACE' is not set in the provided env dictionary.
Variable '$MELTANO_LOAD__TARGET_SCHEMA' is not set in the provided env dictionary.
Variable '$MELTANO_TRANSFORM__PACKAGE_NAME' is not set in the provided env dictionary.
Variable '$MELTANO_EXTRACTOR_NAMESPACE' is not set in the provided env dictionary.
Variable '$MELTANO_LOAD__TARGET_SCHEMA' is not set in the provided env dictionary.
Variable '$MELTANO_TRANSFORM__PACKAGE_NAME' is not set in the provided env dictionary.
Variable '$MELTANO_EXTRACTOR_NAMESPACE' is not set in the provided env dictionary.
Invoking: ['/project/.meltano/transformers/dbt/venv/bin/dbt', '--version']
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to join our friendly Slack community.

Executable 'dbt' could not be found. Transformer 'dbt' may not have been installed yet using `meltano install transformer dbt`, or the executable name may be incorrect.

again, note that meltano is installed and if I try running it manually I'd have no problems with it, only meltano is failing:

root@28ec0fcfb4f8:/project# ls .meltano/transformers/dbt/
venv
root@28ec0fcfb4f8:/project# .meltano/transformers/dbt/venv/bin/dbt --version
Core:
  - installed: 1.10.11
  - latest:    1.10.11 - Up to date!

Plugins:
  - bigquery: 1.10.2 - Up to date!

Are you using the meltano docker container, if not could you share your

Docker

file?

this is the original Dockerfile I was trying to get working

FROM python:3.11

RUN apt-get update && apt-get install git -y

WORKDIR /project
RUN mkdir transformation
RUN mkdir ingestion

# copy meltano project
COPY ingestion/.env ./ingestion
COPY ingestion/meltano.yml ./ingestion

# copy dbt project
COPY transformation/dbt ./transformation
COPY transformation/dbt-sa ./transformation

# configure Meltano project
RUN pip install --upgrade pip
RUN pip install meltano dbt-core dbt-bigquery
WORKDIR /project/ingestion
RUN meltano lock --update --all
RUN meltano install
# ENV PATH="/project/ingestion/.meltano/transformers/dbt/venv/bin:${PATH}"

ENTRYPOINT [ "meltano" ]

however i had the same issue when using the exact same Dockerfile with

FROM meltano/meltano:latest

as a base image. As you can see I usually run meltano lock and install before doing anything so I think the reason why the invokation skips installing dbt is because it was successfully installed before the meltano invoke I thought it might have been either a file permission or dbt not being in the path so i tried including it and checking file permissions, but it didnt really help and files as readable and executable

user@personal-computer:~/meltano-project$ docker run -it --entrypoint bash localmeltano:latest

root@5b8a196d73c1:/project/ingestion# whoami
root
root@5b8a196d73c1:/project/ingestion# ls -la .meltano/transformers/dbt/venv/
total 32
drwxr-xr-x 4 root root 4096 Sep  9 14:06 .
drwxr-xr-x 3 root root 4096 Sep  9 14:05 ..
-rw-r--r-- 1 root root    1 Sep  9 14:05 .gitignore
-rwxrwxrwx 1 root root    0 Sep  9 14:05 .lock
-rw-r--r-- 1 root root   64 Sep  9 14:06 .meltano_plugin_fingerprint
-rw-r--r-- 1 root root   43 Sep  9 14:05 CACHEDIR.TAG
drwxr-xr-x 2 root root 4096 Sep  9 14:06 bin
drwxr-xr-x 3 root root 4096 Sep  9 14:05 lib
lrwxrwxrwx 1 root root    3 Sep  9 14:05 lib64 -> lib
-rw-r--r-- 1 root root  112 Sep  9 14:05 pyvenv.cfg

root@5b8a196d73c1:/project/ingestion# ls -l .meltano/transformers/dbt/venv/bin/dbt 
-rwxr-xr-x 1 root root 336 Sep  9 14:06 .meltano/transformers/dbt/venv/bin/dbt

root@5b8a196d73c1:/project/ingestion# cat /project/ingestion/.meltano/transformers/dbt/venv/bin/dbt
#!/project/ingestion/.meltano/transformers/dbt/venv/bin/python
# -*- coding: utf-8 -*-
import sys
from dbt.cli.main import cli
if __name__ == "__main__":
    if sys.argv[0].endswith("-script.pyw"):
        sys.argv[0] = sys.argv[0][:-11]
    elif sys.argv[0].endswith(".exe"):
        sys.argv[0] = sys.argv[0][:-4]
    sys.exit(cli())

Today I also tried opening a shell in the meltano/meltano:latest image, installed nano, manually copied the meltano.yml and .env files and ran the installation manually, doing the same steps I did when i got the project working on my PC but I still got the same error

Sounds right to me! Glad you got it fixed, we should get that fixed in Meltano so it's easier to tell what is going on

That's a legit and ugly bug, thanks for debugging and reporting back @Daniele Tarek Iaisy! I'll work on a fix.

No problem @Edgar Ramírez (Arch.dev), happy to contribute to the project and thx for the fix!