So I've installed the <Github tap> in order to lea...
# plugins-general
m
So I've installed the Github tap in order to learn how custom taps work in Meltano, and I'm getting a
No such file or directory: '/projects/.meltano/extractors/tap-github/venv/bin/tap-github': '/projects/.meltano/extractors/tap-github/venv/bin/tap-github'
error despite it being within my project directory. I am using the Meltano Docker image and running on Ubuntu if that helps.
d
@michael_cooper That error implies that the plugin wasn't actually installed into
.meltano
(https://gitlab.com/meltano/meltano/-/issues/2223), but that's supposed to happen automatically as part of
meltano add
(https://meltano.com/docs/command-line-interface.html#add). 😕 Did you see anything in your
meltano add
output that suggested something went wrong? Does the entire
.meltano/extractors/tap-github/venv/bin/tap-github
path not exist, or is only the executable missing, for example? You can explicitly install the tap using
meltano install extractor tap-github
, which should solve your problem, but I'm not sure how you got into this situation 😬 Running
meltano install
is usually only necessary after cloning/pulling a project repo, because then the
.meltano
directory will be missing.
m
The path and executable does exist withing
.meltano/extractors/tap-github/venv/bin/tap-github
and I did use
meltano install extractor --custom tap-github
to install it initially. Since it was a singer tap not officially supported by Meltano, I assume I need to use the
--custom
flag?
d
meltano install
doesn't need a
--custom
flag, since it will install all plugins defined in
meltano.yml
, where you've already added
tap-github
🙂
Are you saying that
.meltano/extractors/tap-github/venv/bin/tap-github
already existed before you ran
meltano install
, despite the
No such file or directory: '/projects/.meltano/extractors/tap-github/venv/bin/tap-github'
error message?
m
No. I ran
meltano install
and followed the interactive instructions. I got all the way through the process and got a successful installation.
d
I think you're referring to
meltano add --custom
, not
meltano install
, right?
m
But when I ran the
meltano elt
or
meltano invoke
with the
tap-github
plugin and got the errors.
Correct.
d
Are you still seeing those errors after running
meltano install extractor tap-github
, or are elt and invoke working now?
m
I'll have to get back to you after I try it out (after a meeting)
d
OK, sounds good
m
Okay, I deleted
extractor/tap-github
and removed references from the
meltano.yml
file to start over. Ran
meltano add extractor --custom tap-github
to go through the process of installing, which got me a successful installation. Then ran
meltano install extractor tap-github
which then throws this error:
FileNotFoundError: [Errno 2] No such file or directory: '/projects/.meltano/extractors/tap-github/venv/bin/pip': '/projects/.meltano/extractors/tap-github/venv/bin/pip'
Both
.meltano/extractors/tap-gitub/venv/bin/pip
and ``.meltano/extractors/tap-gitub/venv/bin/tap-github` exists in my directory.
d
@michael_cooper Can you show me how you’re mounting your local directory into the Docker container and how you’re setting the working directory? I think that’s where things are going sideways
m
Of course: 1. cd to my working directory of
Github/learn_meltano
2. run
docker pull meltano/meltano
3. run
docker run -v $(pwd):/projects -w /projects meltano/meltano init learn_meltano
4.
cd learn_meltano
5.
sudo docker run --interactive -v $(pwd):/project -w /project meltano/meltano add --custom extractor tap-github
6.
sudo docker run -v $(pwd):/projects -w /projects meltano/meltano install extractor tap-github
d
@michael_cooper Hmm, that looks right. I thought there may be an issue there because I saw
/projects
in the paths where I was expecting
/project
, but as long as you're consistently mounting the directory and setting the working directory, you should be good. That leaves me quite confused, since if
.meltano/extractors/tap-gitub/venv/bin/pip
exists inside the working directory you're mounting and telling Docker to use as the container's working directory, then why wouldn't the file exist there from the container's perspective as well?
Could you try running bash inside that container, and seeing if you can confirm the existence of
/projects/.meltano/extractors/tap-gitub/venv/bin/tap-github
that way? And then try running
meltano invoke tap-github
from inside that bash shell?
m
Yeah. Running
sudo docker run -it --entrypoint bash meltano/meltano
gets me into the
/projects
, which contains nothing.
Unless I'm running the wrong docker command, which I'm still new to docker as well.
d
Don't forget to mount the directory using
-v $(pwd):/projects -w /projects
😉
Let me know if you'd like to jump on a quick Zoom call so I can help you debug
m
Thank you! And yes, they both exist within the bash shell.
d
I'm out of ideas as far as async text-based debugging goes; wanna join me in https://gitlab.zoom.us/my/gitlab.douwemaan for a few minutes? 🙂
On the call, we realized that the issue arose because the plugin was `add`ed and `install`ed with the project directory mounted at
/project
, but then
elt
and
invoke
were run with the project directory mounted at
/projects
instead, which caused the plugin executables inside the plugin-specific Python virtual environments to fail because they contained hard-coded absolute paths that assumed the project still lived at
/project
.