Hi folks! Do you have any thoughts on making the Meltano plugin installation happen in a PEX and/or portable Python virtualenv? I’m asking as this would greatly simplify installing Meltano using a build tool like Bazel

Hey @Maarten! Wanna say more about what the workflow would look like? (this thread might interest you)

I’m asking as this would greatly simplify installing Meltano using a build tool like Bazel

What are the current pain points you're facing installing Meltano (or plugins)? I've only briefly used bazel in experiments with PyOxidizer, but the experience was not the best mainly because some dependencies don't play well with that type of builds (at the time pendulum, but others too that rely on

__file__

, for example)

Thanks @Edgar Ramírez (Arch.dev) for the issue, subscribed to it! Bazel splits external dependencies (for example the output of meltano install or pip install) and building (for example building a docker image or python executable) into two separate stages. Due to how this works the directory structure is different when installing vs building which results in problems with regular venvs. The problems we had was related to creating a Meltano docker image, both build time and cache invalidation. The build time of the image was high due to Meltano plugins with native extensions which required compilation. Regular cache invalidation because the meltano.yml file is both the dependency and the configuration file. So if any config is changed all of the plugins require reinstalling when building a docker image. Using Bazel (or another build tool) because plugins are installed and cached externally, when building a docker image or running tests we get a lot more cache hits greatly improving the overall feedback cycle. Bonus feature is that plugins are installed/built on-demand, so if developers use Meltano through Bazel they always get Meltano fully installed with the latest plugin versions.