How does package installation function? I'm trying to us the package manager Poetry with a custom tap, and it works fine with only the

setup.py

file, but the added

poetry.lock

and

pyproject.toml

break the installation.

@michael_cooper Meltano creates a Python virtual env for the plugin and then runs

pip install <pip_url>

inside it; nothing more, nothing less. Does that work correctly if you run it manually without Meltano? A package using Poetry for package management and publishing should still be a valid pip package, so I'm not sure why installation would break. Were you seeing any error messages or other specific symptoms of breakage you can share?

Installing locally without the poetry files, i.e. with

-e

, installs fine. If I have those two files within the tap

extract

directory, I get this error:

Installing extractor 'tap-gmail'...
    ERROR: Command errored out with exit status 1:
     command: /projects/.meltano/extractors/tap-gmail/venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/projects/extract/tap-gmail/setup.py'"'"'; __file__='"'"'/projects/extract/tap-gmail/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /projects/extract/tap-gmail/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /projects/.meltano/extractors/tap-gmail/venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/projects/extract/tap-gmail/setup.py'"'"'; __file__='"'"'/projects/extract/tap-gmail/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

Extractor 'tap-gmail' could not be installed: failed to install plugin 'tap-gmail'.

@michael_cooper

ModuleNotFoundError: No module named 'setuptools'

is weird, because

setuptools

is automatically installed in a new venv, and we actually upgrade it to the latest version explicitly: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/venv_service.py#L96 What versions of Python and Meltano are you using?

I'm using

meltano:latest-python3.8

docker image.

Are you installing from pypi or a Git repo? And you're saying that just deleting

poetry.lock

 and 

pyproject.toml

resolves the failure?

Yeah, renaming the files (to effectively delete them) fixes the issue.

And what happens when you directly run

pip install extract/tap-gmail

or

pip install git+url

?

It works fine with

pip install extract/tap-gmail

@michael_cooper That's the weirdest thing 😕 I'd love to figure out what makes the way Meltano runs

pip install

different from you running it directly, and how this specifically trips over those poetry files, with a seemingly unrelated error about

setuptools

.

So the

SETUPTOOLS_USE_DISTUTILS=stdlib

workaround got me past the issue for

setuptools

though now I'm getting the same error for another package used. So it seems like there's still some sort of conflict with pip and the poetry files.

Hmm yeah, seems like poetry is still messing things up.

It does. A team member like's poetry, and he built out the tap using it, but I've had to have a

setup.py

with other taps, so I ended up just adding one.

Interesting, so perhaps we don't need a

setup.py

at all, and need to find Poetry's way of specifying an executable?

I saw that too. Just so I'm not crazy, I can point the

pip_url

to a branch, right? Like

git+https://<url>#my_branch

Yep, if that's the syntax that should work 🙂

So I followed the link you sent on Poetry packages, but for some reason it's still not building the executable.

Is the executable missing just when installing using Meltano, or also when running

pip install

the usual way?

I got it to finally work. Turns out I needed to define the executable plugin within the

pyproject.toml

file.

@michael_cooper All right! Glad you figured it out