Anyone have any idea how to go about specifying a plugin usi Meltano #troubleshooting

Anyone have any idea how to go about specifying a ...

michael_horvath

07/24/2023, 8:52 PM

Anyone have any idea how to go about specifying a plugin using both extras and a git commit? The usual way to do this is provided by the pip documentation (#7 under https://pip.pypa.io/en/latest/cli/pip_install/#examples).

Copy code

-m pip install 'SomePackage[PDF] @ git+<https://git.repo/SomePackage@main#somehash>'

However, I'm having a hard time getting the single quotes to remain when putting this in the pip_url of meltano.json:

Copy code

pip_url: 'SomePackage[PDF] @ git+<https://git.repo/SomePackage@main#somehash>'

Extractor 'SomePackage' could not be installed: failed to install plugin 'SomePackage'. ERROR: Invalid requirement: '@'

edgar_ramirez_mondragon

07/24/2023, 9:37 PM

Hi Michael! Can you try

Copy code

pip_url: "'SomePackage[PDF] @ git+<https://git.repo/SomePackage@main#somehash|https://git.repo/SomePackage@main#somehash>'"

i.e. wrapping it in double quotes

edgar_ramirez_mondragon

07/25/2023, 1:36 AM

(also very very curious if this is actually somehow related to extracting data from a pdf 😅)

chintan_patel

07/25/2023, 12:13 PM

I had similar question https://meltano.slack.com/archives/C01TCRBBJD7/p1690190866439599 Thanks to @Andy Carter I fixed it by defining all the dependencies as below.

Copy code

pip_url: 's3fs>=2023.5.0 fs-s3fs>=1.1.1 git+<https://github.com/MeltanoLabs/tap-universal-file.git>'

chintan_patel

07/25/2023, 12:15 PM

With

It will not work because in cli source code It just splitting the input by space. Check this. https://github.com/meltano/meltano/blob/987c71c0ff3348b79c8151caa7fdbf84e8a49757/src/meltano/core/plugin_install_service.py#L505

michael_horvath

07/25/2023, 12:44 PM

Unfortunately not Edgar, that ends up with:

ERROR: Invalid requirement: "'SomePackage[PDF]"

Looking at the code Chintan pointed out, it doesn't look like there is currently a way to make this work. The pip_url doesn't expect quotes. I think it requires a pull request to use https://docs.python.org/3/library/shlex.html instead

michael_horvath

07/25/2023, 1:39 PM

Also to your side question, I'm not working with PDFs. I was just trying to keep the example from pypa consistent, I am building up tooling to extract from XML and have a branch that I'm not ready to submit a pull request to spreadsheets anywhere. Has extras requirements for the XML dependencies

edgar_ramirez_mondragon

07/25/2023, 4:58 PM

Ok, gotcha https://github.com/meltano/meltano/pull/7947

2 Views

Open in Slack

Previous Next