Is there any way to have meltano use the pip packages installed in the system, rather than resorting to installing them into their own virtual environments? I’m trying to containerize my meltano project and thought it would be easier to manage all the required python packages in one place.

Meltano is very tightly coupled to venvs and where they exist. I containerize meltano and personally would really recommend just adding

RUN meltano install

or

RUN meltano install extractor ... && meltano install loader ... && ...

To the dockerfile. The venvs are really lightweight in the grand scheme of things

rather than resorting to installing them into their own virtual environments?

That usually creates more headaches than it solves, because of dependency conflicts between singer taps, targets, dbt, etc. @sterling_paramore I’m curious about which benefits you are looking for by installing all the packages in a single place (e.g. build time, image size, security)

For local dev, I always mount my project into the container, and I can’t figure out how to do

RUN meltano install

in that case. If I could install the pip dependencies in a directory outside of my code repo, that might work, but that feature isn’t currently available. My concern here is mostly about building a container I can deploy and also use as a dev environment.

How do you build your container?

FROM meltano/...

WORKDIR /project
COPY . .

RUN meltano install

👆 this will give you a container with meltano ready to go! which you could enter into overriding the entrypoint to /bin/bash if you want to do some work out of it Using a mount, your mounted files can override the workdir files but that could be intentional if you have config locally you want to be the source of truth -- so maybe it solves for your needs

docker run -it -v ./:/project -- /bin/bash

definitely some pseudocode going on above 😄 but does that make sense? hope it helps!

So, one way is to simply have the tap,target in the path(which already happens when installing them). Then, just mention its cli name in the executable. It will be enough.

You know what….. I hadn’t thought of the fact that I could run

meltano install

in the

Dockerfile

, and then ALSO run it as a separate task to initialize my local dev environment after mounting my code to the container. That seems like it will work ok, despite a little duplication of effort. Thanks Alex!