Our meltano project has so far only utilized public plugins Meltano #troubleshooting

Our meltano project has so far only utilized publi...

Matt Menzenski

01/31/2024, 2:49 PM

Our meltano project has so far only utilized public plugins but we now have a need to use some private plugins. We have a private PyPi mirror (Artifactory) and I’d like to install the private plugins from there. I see in the docs a section on “*Environment variable expansion within `pip_url`*”:

Another use for this is to supply credentials for a private Python package index:

Copy code

pip_url: "https://${NEXUS_USERNAME}:${NEXUS_PASSWORD}@nexus.example.com/simple"

Actually maybe https://docs.meltano.com/guide/plugin-management#installing-plugins-from-a-custom-python-package-index-pypi is the better documentation for this.

✅ 1

Matt Menzenski

01/31/2024, 2:53 PM

I ended up answering my own question here, posting this all for some visibility into this (since I didn’t find any examples in GitHub of people installing plugins from private PyPi repositories. Tracing the code execution path for

meltano install

winds up here: - whatever is provided in the

pip_url

is tokenized here and passed directly to

pip install

. This worked for me:

Copy code

pip_url: >-
  tap-my-custom-plugin==0.1.0
  --extra-index-url https://${MY_USERNAME}:${MY_PASSWORD}@my-artifactory-host.com/artifactory/api/pypi/pypi/simple
  --trusted-host <http://my-artifactory-host.com|my-artifactory-host.com>

🔥 1

marcin_wojciechowski

01/31/2024, 3:44 PM

Thanks for sharing this Matt, we may need to private plugin repositories as well. Just out of interest, what's your use case to use private vs. open source plugin?

Matt Menzenski

01/31/2024, 4:05 PM

I do hope to be able to open-source this particular plugin in the future. But it requires a lot of custom configuration that’s very specific to my organization and it’s easier in the short term to just define that configuration inside the tap and start using it. This particular tap hits an ElasticSearch /search API endpoint so there’s a lot of JSON configuration of the POST body that’s passed to the endpoint. I asked a question around the handling of JSON configuration files in https://meltano.slack.com/archives/C068YBV6KEK/p1706636806228989 - this is the same tap. If I can figure out a good way to define the configuration in JSON files and allow consumers to provide their own configuration files, I would try to opensource the tap. But today it’s more important that I get it up and running for our internal needs.

👍 1

visch

01/31/2024, 5:39 PM

https://docs.meltano.com/guide/plugin-management/#installing-plugins-from-a-custom-python-package-index-pypi

visch

01/31/2024, 5:39 PM

ARG PIP_INDEX_URL=<your_custom_pypi_url>

gets you there as well and might be more what you're after

➕ 1

Edgar Ramírez (Arch.dev)

01/31/2024, 7:50 PM

+1 to Derek's suggestion. Setting the env var

Copy code

PIP_EXTRA_INDEX_URL=...

Copy code

PIP_INDEX_URL=...

wherever

meltano install

runs should do the trick.

visch

01/31/2024, 7:51 PM

I think Edgar is the one that documented this so 🙌 edgar 😄

melty bouncy 1

Matt Menzenski

01/31/2024, 9:01 PM

For the time being I am probably going to stick with the single plugin pip_url approach because it lets me reference user and password environment variables that already exist on developer machines in my org (so I can be confident that my addition of a private dependency isn’t going to break anyone’s local testing, etc). I’m interested in the

PIP_EXTRA_INDEX_URL

PIP_INDEX_URL

option but that’ll take more coordination to adopt in a way that won’t impact anyone.

👍 2

4 Views

Open in Slack

Previous Next