Hi! Have you considered building Meltano as conda package?

Hi @jan_maly! Thanks for asking 😄. No, I don't think we have. I'm not all too familiar with the conda ecosystem so I have to ask: aren't the PyPI packages installable by conda? Or maybe I'm missing something 😅

The motivation: We are working on a very slim and performant Docker container with Meltano. We have run into several issues because of some missing system libraries required by plugins. It is hard to install those in a Python environment with pip (it is limited to Python). Conda could solve the issues: Conda is a cross-platform package and environment manager that installs and manages conda packages. Conda packages are binaries. There is never a need to have compilers available to install them. Additionally conda packages are not limited to Python software. Not familiar with creating conda packages, but they are very popular in the Data Science community.

@jan_maly thanks for sharing!

We are working on a very slim and performant Docker container with Meltano. We have run into several issues because of some missing system libraries required by plugins. It is hard to install those in a Python environment with pip (it is limited to Python).

One avenue I have explored in the past that might work for this use case too is having a multi-stage build. The first stage has all the compilers and system dependencies required to build the python packages. The second stage is a more lightweight one without the compilers that just copies over the virtual environment(s) from the first one. There's an example here that might be a good starting point: https://pythonspeed.com/articles/multi-stage-docker-python/

Conda is a cross-platform package and environment manager that installs and manages conda packages. Conda packages are binaries. There is never a need to have compilers available to install them. Additionally conda packages are not limited to Python software.

I think publishing a Meltano wheel to PyPI (we're only shipping a .tar.gz sdist at the moment) solves this problem. That would require us to build wheels for multiple target systems though (windows, macOS, linux, etc.), but I'm pretty sure conda packaging would impose a similar requirement. cc @aaronsteers

@edgar_ramirez_mondragon - Thanks for this detailed response. @jan_maly - Would you be interested in logging an issue for us on the potential improvements here - perhaps with additional details on the build process and specific dependencies/delays you are running into in the status quo? I am not sure if conda publish or publish-as-wheel is the best investment but certainly sounds like there's some improvements we could make here. 👍

Our issues are already solved. First, I will look into "build conda" package to evaluate the time investment. I might be able to do that on my own and share 🙂