Hi Everyone. We are just beginning to use Meltano for our ELT work and it’s been quite interesting so far. Have a question as I am getting set up here. We are in a custom docker container and have done a

meltano install

which is connecting to our a Postgres backend like it’s supposed to. But when it tries to install the 3 plug ins inside the container, we get this error: I was wondering if anyone knows why this happens. It works perfectly outside the container inside a virtualenv but not inside. I have only just started using Meltano, so please let me know if you need more information

Installing 3 plugins...
Installing loader 'target-mysql'...
Loader 'target-mysql' could not be installed: could not create of the virtualenv for 'loaders/target-mysql'

Installing loader 'target-postgres'...
Loader 'target-postgres' could not be installed: could not create of the virtualenv for 'loaders/target-postgres'

Installing extractor 'tap-s3-csv'...
Extractor 'tap-s3-csv' could not be installed: could not create of the virtualenv for 'extractors/tap-s3-csv'

Installed 0/3 plugins
Failed to install plugin(s)

@rohith_desikan First thing to check: are you using the most recent version of Meltano?

meltano --version

We are using meltano version 1.72.0. There has been some issue with parallel installs, we are getting some kind of async error, which is why we are using a slightly older 1.

Ah OK, that’s not good! Would you mind separately starting a thread/issue about that parallel install/async problem, or see if one has already been created?

Yeah, let’s see if I can replicate this async issue with a 1.77 install

OK, this was reported recently but it doesn’t look like an issue was created: https://meltano.slack.com/archives/C01TCRBBJD7/p1623801856256800

Yep, he’s my colleague. I was trying to find his message. I will create the issue on Gitlab.

Ah perfect, thanks

It is based on our custom Docker image. This is our base image:

FROM ubuntu:20.04
ENV TZ=America
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

# Install required Ubuntu packages
# Install sudo
RUN apt-get update && \
      apt-get -y install sudo \
      tzdata
    
# Install python packages
RUN apt-get update && \
    apt-get install -y -q \
    curl \
    gcc \
    gnupg \
    jq \
    libpq-dev \
    netcat \
    python3 \
    python3-distutils \
    python3-pip \
    vim

RUN sudo apt-get install -y -q \
    postgresql \
    postgresql-contrib

RUN echo en_US.UTF-8 UTF-8 > /etc/locale.gen
RUN locale-gen en_US.UTF-8

RUN apt-get clean && rm -rf /var/lib/apt/lists/*

# Symlink python to python3
RUN ln -s /usr/bin/python3 /usr/bin/python

# We copy just the requirements.txt first to leverage Docker cache
COPY requirements.txt /opt/wg/requirements.txt

# Install pip
RUN curl <https://bootstrap.pypa.io/get-pip.py> -o /tmp/get-pip.py && \
    python /tmp/get-pip.py && \
    pip3 install --upgrade pip && \
    pip3 install -r /opt/wg/requirements.txt
    
RUN rm -f /tmp/get-pip.py

# Set the working directory
WORKDIR /opt/wg

meltano==1.72.0 lives in the

requirements.txt

file. The rest of the Dockerfile looks like below. We have a local Postgres container running, for which I export a MELTANO_DATABASE_URI environment var and the I can see the meltano migrations working as they should.

FROM <ECR_PRIVATE_REPO>

RUN apt-get update && \
    apt-get install -y -q \
    git 

# Set up uvicorn service (FastAPI)
ENV USER=uvicorn
ENV UID=10001
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/home/${USER}" \
    --shell "/bin/bash" \
    --uid "${UID}" \
    "${USER}"

# FOLDER SETUP FOR EXAMPLE (This just copies repo files/folders into the container)
RUN mkdir -p /opt/wg/bin
ADD bin /opt/wg/bin

WORKDIR /opt/wg

Which version of Python are you getting?

Python 3.8

OK good

You mean add them to my path? Same error.

I meant, what do you get when you run

python -m venv .meltano/loaders/target-postgres/venv

directly in the container, instead of going through

meltano install

?

Ah I see, looks like I need another package. Let’s install and see what happens:

The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

Aah, interesting!

OMG it worked. Thank you so much. Here it is:

Installing 3 plugins...
Installing loader 'target-mysql'...
Installed loader 'target-mysql'

Installing loader 'target-postgres'...
Installed loader 'target-postgres'

Installing extractor 'tap-s3-csv'...
Installed extractor 'tap-s3-csv'

Installed 3/3 plugins

Thank you @rohith_desikan! Glad I was able to help you figure it out

Hey btw, when I added that

apt-get install python3-venv

command, meltano version 1.77.0 installs all the plug ins. I no longer get that coroutine error.

@rohith_desikan Oh, great! I think the coroutine error is raised in the “parallel install” error handling, so you would only see it if there was an error to handle.

Ok sure

@rohith_desikan Thanks, this is enough to start!