So I've installed the Github tap in order to learn how custom taps work in Meltano, and I'm getting a

No such file or directory: '/projects/.meltano/extractors/tap-github/venv/bin/tap-github': '/projects/.meltano/extractors/tap-github/venv/bin/tap-github'

error despite it being within my project directory. I am using the Meltano Docker image and running on Ubuntu if that helps.

@michael_cooper That error implies that the plugin wasn't actually installed into

.meltano

(https://gitlab.com/meltano/meltano/-/issues/2223), but that's supposed to happen automatically as part of

meltano add

(https://meltano.com/docs/command-line-interface.html#add). 😕 Did you see anything in your

meltano add

output that suggested something went wrong? Does the entire

.meltano/extractors/tap-github/venv/bin/tap-github

path not exist, or is only the executable missing, for example? You can explicitly install the tap using

meltano install extractor tap-github

, which should solve your problem, but I'm not sure how you got into this situation 😬 Running

meltano install

is usually only necessary after cloning/pulling a project repo, because then the

.meltano

directory will be missing.

The path and executable does exist withing

.meltano/extractors/tap-github/venv/bin/tap-github

and I did use

meltano install extractor --custom tap-github

to install it initially. Since it was a singer tap not officially supported by Meltano, I assume I need to use the

--custom

flag?

meltano install

doesn't need a

--custom

flag, since it will install all plugins defined in

meltano.yml

, where you've already added

tap-github

🙂

No. I ran

meltano install

and followed the interactive instructions. I got all the way through the process and got a successful installation.

I think you're referring to

meltano add --custom

, not

meltano install

, right?

But when I ran the

meltano elt

or

meltano invoke

with the

tap-github

plugin and got the errors.

Are you still seeing those errors after running

meltano install extractor tap-github

, or are elt and invoke working now?

I'll have to get back to you after I try it out (after a meeting)

OK, sounds good

Okay, I deleted

extractor/tap-github

and removed references from the

meltano.yml

file to start over. Ran

meltano add extractor --custom tap-github

to go through the process of installing, which got me a successful installation. Then ran

meltano install extractor tap-github

which then throws this error:

FileNotFoundError: [Errno 2] No such file or directory: '/projects/.meltano/extractors/tap-github/venv/bin/pip': '/projects/.meltano/extractors/tap-github/venv/bin/pip'

Both

.meltano/extractors/tap-gitub/venv/bin/pip

and ``.meltano/extractors/tap-gitub/venv/bin/tap-github` exists in my directory.

@michael_cooper Can you show me how you’re mounting your local directory into the Docker container and how you’re setting the working directory? I think that’s where things are going sideways

Of course: 1. cd to my working directory of

Github/learn_meltano

2. run

docker pull meltano/meltano

3. run

docker run -v $(pwd):/projects -w /projects meltano/meltano init learn_meltano

4.

cd learn_meltano

5.

sudo docker run --interactive -v $(pwd):/project -w /project meltano/meltano add --custom extractor tap-github

6.

sudo docker run -v $(pwd):/projects -w /projects meltano/meltano install extractor tap-github

@michael_cooper Hmm, that looks right. I thought there may be an issue there because I saw

/projects

in the paths where I was expecting

/project

, but as long as you're consistently mounting the directory and setting the working directory, you should be good. That leaves me quite confused, since if

.meltano/extractors/tap-gitub/venv/bin/pip

exists inside the working directory you're mounting and telling Docker to use as the container's working directory, then why wouldn't the file exist there from the container's perspective as well?

Yeah. Running

sudo docker run -it --entrypoint bash meltano/meltano

gets me into the

/projects

, which contains nothing.

Don't forget to mount the directory using

-v $(pwd):/projects -w /projects

😉

Thank you! And yes, they both exist within the bash shell.

I'm out of ideas as far as async text-based debugging goes; wanna join me in https://gitlab.zoom.us/my/gitlab.douwemaan for a few minutes? 🙂