Hey folks I m having some trouble managing plugins wondering Meltano #troubleshooting

Hey folks! I'm having some trouble managing plugi...

Matt Lam

02/15/2025, 1:40 AM

Hey folks! I'm having some trouble managing plugins, wondering if anyone else ran into similar behaviour... So I'm trying to use a tap (

tap-gong

) that only supports Python versions up to 3.8...which is EOL as of late last year. I can successfully install and run the tap using the --force-install and --no-install flags on

meltano add

and

meltano run

respectively:

Copy code

> meltano add extractor tap-gong --force-install
> meltano run tap-gong target-bigquery --no-install

However, if I stray from that happy path, Meltano doesn't seem to respect the --force-install and --no-install flags. For instance,

meltano invoke tap-gong --no-install

after adding the plugin gives me console spew that looks like:

Copy code

...
2025-02-15T01:38:27.389693Z [info     ] Installing extractor 'tap-gong'
2025-02-15T01:38:27.390744Z [debug    ] Packages for 'extractors/tap-gong' have changed so performing a clean install.
2025-02-15T01:38:27.906315Z [debug    ] Removed old virtual environment for 'extractors/tap-gong'
...

For what it's worth,

--force-install

also triggers an install (of course), but then the pip logs tell me that installation fails because of version constraints, so

--force-install

isn't really being respected either. I checked the meltano GitHub project for existing issues regarding these flags but couldn't find anything. Anyone else run into similar problems or have any advice? Thanks!

✅ 1

Reuben (Matatika)

02/15/2025, 6:18 PM

--no-install

is in fact installing plugins, then that is a bug. Although, I'm curious why you are using those options in this context? Does the following work?

Copy code

meltano add --python python3.8 extractor tap-gong

# subsequent command invocations should not need to install plugin again
meltano run ...
meltano invoke ...

Reuben (Matatika)

02/16/2025, 8:23 PM

Just tested this myself and realised it's because arguments/options specified after

meltano invoke <plugin>

are passed to the plugin executable, not Meltano. So

meltano invoke <plugin> --no-install

will still try and install the plugin

Copy code

reuben@reuben-Inspiron-14-5425:/tmp/p$ meltano invoke tap-gong --no-install
2025-02-16T20:09:50.616625Z [info     ] Environment 'dev' is active   
2025-02-16T20:09:50.764298Z [info     ] Installing extractor 'tap-gong'
2025-02-16T20:10:07.076203Z [info     ] Installed extractor 'tap-gong'
Usage: tap-gong [OPTIONS]
Try 'tap-gong --help' for help.

Error: No such option: --no-install

but

meltano invoke --no-install <plugin>

will not

Copy code

reuben@reuben-Inspiron-14-5425:/tmp/p$ meltano invoke --no-install tap-gong
2025-02-16T20:12:29.154732Z [info     ] Environment 'dev' is active   
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Executable 'tap-gong' could not be found. Extractor 'tap-gong' may not have been installed yet using `meltano install extractor tap-gong`, or the executable name may be incorrect.

Pretty sure this is

invoke

-specific.

💡 1

Matt Lam

02/17/2025, 7:03 PM

Thanks for getting back to me on this! I wasn't sure

meltano add --python python3.8 extractor tap-gong

would work for me because I'm running this in a conda environment, and it seem hard (not possible?) to install multiple python executables in a single conda environment. But what you mention about the ordering of the arguments makes sense, there are obviously cases where one needs to pass arguments to the executable after all. I'll try this when I get a change in the next couple of days. Thanks again!

Reuben (Matatika)

02/20/2025, 11:25 AM

If you're unable to reference a Python 3.8 executable and

--force-install

doesn't work, then I imagine you aren't going to get much further without opening an issue to the tap repo or forking it and making the change yourself (looks like there is repo activity so I would suggest following up with a PR in that case). Regarding positional options: I'm not exactly sure of the convention, but there might be an optimisation that can be made to optimistically treat options specified after

invoke

as for Meltano first, or otherwise employ the

--

"end of command options" delimiter to treat everything after as for the plugin only. Then

--no-install

in the position you were providing it could work:

Copy code

meltano invoke <tap> --no-install -- (tap args/options)

I'm sure there's some nuance to it - @Edgar Ramírez (Arch.dev) may have a better understanding.

Reuben (Matatika)

02/20/2025, 11:35 AM

you aren't going to get much further without opening an issue to the tap repo

Never mind, they have issues disabled for some reason...? Looks like others have fixed in the meantime if you want to try that version: https://github.com/Hack-Attack/tap-gong

meltano.yml

Copy code

- name: tap-gong
    variant: symon-ai
    pip_url: git+<https://github.com/Hack-Attack/tap-gong.git>

Copy code

meltano install --clean extractor tap-gong

Matt Lam

02/20/2025, 6:09 PM

just to close the loop -- ordering the flags appropriately (ie.

meltano invoke --no-install tap-gong

) works for me! So in this way, I can force-install at the start of my workflow and then prevent meltano from trying to re-install on subsequent commands. Thanks!

👍 2

4 Views

Open in Slack

Previous Next