https://linen.dev logo
Join Slack
Powered by
`meltano install` must be run inside a Meltano pro...
# docker
c
meltano install
must be run inside a Meltano project. I'm building a docker container and getting the error above. I create a project directory with 
meltano init
then I set that project directory as my WORKDIR copy my 
meltano.yml
and 
RUN meltano install
It seems like I'm in the Meltano project - any thoughts? Here's the whole Dockerfile
Copy code
ARG PYTHON_IMAGE=python:3.10-slim
FROM $PYTHON_IMAGE


ARG MELTANO_PROJECT_NAME=vs
ARG MELTANO_PROJECT_ROOT=workspaces

RUN apt update
# build-essential helps build meltano plugins (specifically target-postgres)
RUN apt install -y build-essential python3-venv
# git is required for installing meltano plugins
RUN apt install -y git
# libpq-dev is required for target-postgres
RUN apt install -y libpq-dev

# Install any additional requirements
RUN /usr/local/bin/python -m pip install --upgrade pip
COPY requirements.txt ./
RUN pip install -r ./requirements.txt 

# pipx is preferred for using meltano
RUN pipx install meltano
ENV PATH "$PATH:/root/.local/bin"

# create the meltano project folder
WORKDIR /$MELTANO_PROJECT_ROOT
RUN echo "PWD is $PWD"
RUN meltano init "$MELTANO_PROJECT_NAME"

# install the Meltano plugins as specified in meltano.yml
# TODO - this could be a mounted folder so the yml can be iterated rapidly
WORKDIR /$MELTANO_PROJECT_ROOT/$MELTANO_PROJECT_NAME
RUN echo "PWD is>: $PWD"
COPY meltano.yml ./ 
RUN meltano install
v
https://hub.docker.com/r/meltano/meltano already exists and did a lot of this work. https://github.com/meltano/meltano/blob/main/docker/meltano/Dockerfile has a file you can use to compare what meltano did vs what you're trying to do
To debug this if I were you is I'd run a bash shell and then run ls, and 
meltano init
myself to see what's going on
RUN meltano init "$MELTANO_PROJECT_NAME"
looks sus. as does 
COPY meltano.yml ./
but I"d just debug them in the shell instead of guessing
m
I’ve had good success running
Copy code
meltano add files files-docker
and then tweaking the Dockerfile that is generated
c
First, Thanks folks! I did look at the meltano/meltano image first, but it seems to be focused on running meltano-ui rather than extracting and loading. Then I rolled my own using bash inside the container to write the commands and then transferred them to the Dockerfile. I'm quite green with writing Dockerfiles, but I found a post here about @jacob_matson that referenced a repo he has where I found a fantastic example, which moved me along considerably. The other thing that I'll leave here for future me is the 
--progress=plan
argument for docker build is incredibly helpful as it lets me review the output from any commands, and essentially add print statements like 
RUN echo "PWD is $PWD"
to help with debugging. That essentially eliminated my need to run bash in the container.
To your question about 
meltano init
the output looks completely normal with the big meltano banner and 
Your project has been created!
.
regarding 
copy meltano.yml ./
I was assuming this worked, but because I couldn't figure out how to echo the contents so I could view it at build time, I dove into the container shell. What I found (or didn't find) has me confused. docker build shows me that the pwd is /workspaces in one case and /workspaces/vs in another, but from inside the container I can't find those directories. I'm going to delete the container and image and start from scratch to make sure I don't have something cached that is messing with me. Thanks for asking a thought-provoking question!
when I went to delete all the containers and images I figured out that I had two different images and my initial investigation was with the wrong image. That solved, I did find my /workspaces/vs folder as expected. And when I'm in the shell in the container and run 
meltano install
it works perfectly. But when I run the command in the docker build I get the same error: 
'meltano install' must be run inside a Meltano project.
I swapped out the way I was installing meltano via pipx and I'm following @jacob_matson's example more closely. Init still works and everything's good, right up to the copy meltano.yml command, but I get the same error about needing to be inside a meltano project - and I can confirm that I am in the /vs folder that was created by 
meltano init
. Now I'm wondering if there are some env variables that set when meltano init is run, that are not present several commands later when I get to the install command...
nope - from within the container shell, the env vars are the same before and after running 
meltano init
.
Here's my updated Dockerfile
Copy code
ARG PYTHON_IMAGE=python:3.10-slim

FROM $PYTHON_IMAGE

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

ARG MELTANO_PROJECT_NAME=vs
ARG MELTANO_PROJECT_ROOT=workspaces

RUN apt update
# build-essential helps build meltano plugins (specifically target-postgres)
RUN apt install -y build-essential python3-dev python3-venv
# git is required for installing meltano plugins
RUN apt install -y git
# libpq-dev is required for target-postgres
RUN apt install -y libpq-dev

# Install any additional requirements
RUN /usr/local/bin/python -m pip install --upgrade pip
# psycopg2 required for target-postgres
RUN pip install psycopg2-binary==2.9.5

RUN pip install --no-cache-dir meltano==2.10.0

# create the meltano project folder
WORKDIR /$MELTANO_PROJECT_ROOT
RUN echo "PWD is $PWD"
RUN meltano init "$MELTANO_PROJECT_NAME"

# install the Meltano plugins as specified in meltano.yml
WORKDIR /$MELTANO_PROJECT_ROOT/$MELTANO_PROJECT_NAME
RUN echo "PWD is: $PWD"
COPY meltano.yml ./ 
RUN meltano install
I'm on a good path - just copied Jacob's Dockerfile and meltano.yml and they work. Now I'm going to make incremental changes as I morph it into what I need. I should be able to figure out where I was off the rails.
ok, this one is a bit crazy - re-adding my own code step-by-step to the Dockerfile I ran into one line that causes 
RUN meltano install
to fail. drum roll please.... 
ARG MELTANO_PROJECT_ROOT=workspaces
And the problem is the arg name, not the value. In the following Dockerfile, if I enable the arg it will fail, but disabled the meltano install command works.
Copy code
FROM python:3.9

WORKDIR /workspaces/vs

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

RUN apt-get update && apt-get install -y \
  gcc \
  g++ \
  git \
  make \
  python3-dev \
  python3-venv \
  unzip \
  wget \
  && rm -rf /var/lib/apt/lists/* \
  && pip install --no-cache-dir meltano==2.10.0

COPY meltano.yml ./ 

# ARG MELTANO_PROJECT_ROOT=workspaces

RUN meltano --environment=docker install
I thought perhaps there was a conflict with an env var used by meltano, or some other variable name, but I can't find it on github. So, this feels like a bug, but I've got no idea what's causing it yet. https://github.com/meltano/meltano/issues/7106
2 Views
Open in Slack
PreviousNext
Powered by