The misleading error message is a known issue, by the way: https://gitlab.com/meltano/meltano/-/issues/2573

```meltano --log-level=debug invoke airflow scheduler -D [2021-09-03 224943,757] [2345|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x7f1ce82f2b20>@sqlite:////home/ubuntu/meltano_proj_repo/.meltano/meltano.db [2021-09-03 224943,803] [2345|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): www.meltano.com:443 [2021-09-03 224944,375] [2345|MainThread|urllib3.connectionpool] [DEBUG] https://www.meltano.com:443 "GET /discovery.yml HTTP/1.1" 200 115895 [2021-09-03 224944,860] [2345|MainThread|root] [ERROR] The

discovery.yml

manifest received from https://www.meltano.com/discovery.yml could not be parsed. [2021-09-03 224944,860] [2345|MainThread|root] [DEBUG] [Errno 13] Permission denied: '/home/ubuntu/meltano_proj_repo/.meltano/cache/discovery.yml' [2021-09-03 224944,892] [2345|MainThread|root] [DEBUG] Invoking: ['/home/ubuntu/meltano_proj_repo/.meltano/orchestrators/airflow/venv/bin/airflow', '--help'] [2021-09-03 224944,892] [2345|MainThread|root] [DEBUG] Env: {'SHELL': '/bin/bash', 'LC_ADDRESS': 'C.UTF-8', 'LC_NAME': 'C.UTF-8', 'LC_MONETARY': 'C.UTF-8', 'PWD': '/home/ubuntu/meltano_proj_repo', 'LOGNAME': 'ubuntu', 'XDG_SESSION_TYPE': 'tty', 'MOTD_SHOWN': 'pam', 'HOME': '/home/ubuntu', 'LC_PAPER': 'C.UTF-8', 'LANG': 'C.UTF-8', 'LS_COLORS': 'rs=0di=01;34ln=01;36mh=00pi=40;33so=01;35do=01;35bd=40;33;01cd=40;33;01or=40;31;01mi=00su=37;41sg=30;43ca=30;41tw=30;42ow=34;42st=37;44ex=01;32*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'SSH_CONNECTION': '10.22.143.1 45576 10.22.143.76 22', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'TERM': 'xterm-256color', 'LC_IDENTIFICATION': 'C.UTF-8', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'USER': 'ubuntu', 'SHLVL': '1', 'LC_TELEPHONE': 'C.UTF-8', 'LC_MEASUREMENT': 'C.UTF-8', 'XDG_SESSION_ID': '415', 'XDG_RUNTIME_DIR': '/run/user/1000', 'SSH_CLIENT': '10.22.143.1 45576 22', 'LC_TIME': 'C.UTF-8', 'XDG_DATA_DIRS': '/usr/local/share/usr/share/var/lib/snapd/desktop', 'PATH': '/home/ubuntu/meltano_proj_repo/.meltano/orchestrators/airflow/venv/bin/usr/local/sbin/usr/local/bin/usr/sbin/usr/bin/sbin/bin/usr/games/usr/local/games:/snap/bin', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'SSH_TTY': '/dev/pts/0', 'LC_NUMERIC': 'C.UTF-8', 'BASH_FUNC_juju_machine_lock%%': '() { for agent in $(ls /var/lib/juju/agents);\n do\n juju_agent machinelock $agent 2> /dev/null;\n done\n}', 'BASH_FUNC_juju_statetracker_report%%': '() { juju_agent debug/pprof/juju/state/tracker?debug=1\n}', 'BASH_FUNC_juju_agent_call%%': '() { local agent=$1;\n shift;\n juju-introspect --ag…

let me double check if my juju charm does

meltano add orchestrator airflow

thank you for the help finding the verbose mode @douwe_maan, it turned out I needed to not just add airflow but debug my tap.. which was not present in the environment. I use Juju to instantiate meltano in a container so.. the container needed to be setup properly Thanks again!

hmmm on second check.. when I start it's still failing.. let me re-read here your post and see if I missed something

ahhh i see my problem

ubuntu@juju-2dd159-150:~/meltano_proj_repo$ meltano install 
[2021-09-04 11:26:56,249] [1192|MainThread|root] [ERROR] The `discovery.yml` manifest received from <https://www.meltano.com/discovery.yml> could not be parsed.
Extractor 'tap-ibkr' is not known to Meltano

trying to patch this up but.. seems to fail to create the virtualenv

```meltano --log-level=debug add --custom extractor tap-ibkr [2021-09-04 114113,125] [1384|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x7f07901fddf0>@sqlite:////home/ubuntu/meltano_proj_repo/.meltano/meltano.db Adding new custom extractor with name 'tap-ibkr'... Specify the plugin's namespace, which will serve as the: - identifier to find related/compatible plugins - default database schema (

load_schema

extra), for use by loaders that support a target schema Hit Return to accept the default: plugin name with underscores instead of dashes (namespace) [tap_ibkr]: Specify the plugin's

pip install

argument, for example: - PyPI package name: tap-ibkr - Git repository URL: git+https://gitlab.com/meltano/tap-ibkr.git - local directory, in editable/development mode: -e extract/tap-ibkr - 'n' if using a local executable (nothing to install) Default: plugin name as PyPI package name (pip_url) [tap-ibkr]: /home/ubuntu/meltano_tap_ibkr/tap-ibkr Specify the plugin's executable name Default: name derived from

pip_url

(executable) [tap-ibkr]: Specify the tap's supported Singer features (executable flags), for example: `catalog`: supports the

--catalog

flag `discover`: supports the

--discover

flag `properties`: supports the

--properties

flag `state`: supports the

--state

flag To find out what features a tap supports, reference its documentation or try one of the tricks under https://meltano.com/docs/contributor-guide.html#how-to-test-a-tap. Multiple capabilities can be separated using commas. Default: no capabilities (capabilities) [[]]: Specify the tap's supported settings (

config.json

keys) Multiple setting names (keys) can be separated using commas. A setting kind can be specified alongside the name (key) by using the

:

delimiter, e.g.

port:integer

to set the kind

integer

for the name

port

Supported setting kinds: string | integer | boolean | date_iso8601 | email | password | oauth | options | file | array | object | hidden - Credentials and other sensitive setting types should use the password kind. - If not specified, setting kind defaults to string. - Nested properties can be represented using the

.

separator, e.g.

auth.username

for

{ "auth": { "username": value } }

. - To find out what settings a tap supports, reference its documentation. Default: no settings (settings) [[]]: [2021-09-04 114149,254] [1384|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): www.meltano.com:443 [2021-09-04 114149,826] [1384|MainThread|urllib3.connectionpool] [DEBUG] https://www.meltano.com:443 "GET /discovery.yml HTTP/1.1" 200 115895 [2021-09-04 114150,324] [1384|MainThread|root] [ERROR] The

discovery.yml

manifest received from https://www.meltano.com/discovery.yml could not be parsed. [2021-09-04 114150,324] [1384|MainThread|root] [DEBUG] [Errno 13] Permission denied: '/home/ubuntu/meltano_proj_repo/.meltano/cache/discovery.yml' Added extractor 'tap-ibkr' to your Meltano project [2021-09-04 114150,368] [1384|MainThread|asyncio] [DEBUG] Using selector: EpollSelector Installing extractor 'tap-ibkr'... [2021-09-04 114150,369] [1384|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'extractors/tap-ibkr' [2021-09-04 114150,369] [1384|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/tap-ibkr Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/tap-ibkr' [2021-09-04 114150,369] [1384|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'extractors/tap-ibkr' Extractor 'tap-ibkr' could not be installed: could not create the virtua…

im quite stuck now.. but basically I want to simply run a private tap so I can develop it and improve it..

- name: tap-ibkr
    pip_url: <git+ssh://git@github.com/MYGROUP/REPO.git>
    executable: /home/ubuntu/meltano_tap_repo/tap-ibkr/tap-ibkr.sh

but this doesn't seem to be enough and then I goto try re-adding or re-installing it in the yaml but getting stuck there with what seems like permission issues?

it's a private repo too.. maybe thats the issue?

the code is already cloned but.. unsure maybe it's unhappy with somethign because it's a SSH git url?

ahhh finally I figured it all out

turns out... Meltano doesn't install without being used in a virtualized python .. EVEN in a containerized environment.. it just fails fails fails.. BUT as soon as I added a basic venv.. inside my container.. all is smooth and going well

ubuntu@juju-2dd159-155:~/meltano_proj_repo$ python -m venv .venv/meltano
ubuntu@juju-2dd159-155:~/meltano_proj_repo$ source .venv/meltano/bin/activate
(meltano) ubuntu@juju-2dd159-155:~/meltano_proj_repo$ meltano install
Installing 5 plugins...
Installing file bundle 'airflow'...
Installing orchestrator 'airflow'...
Installing loader 'target-postgres'...
Installing extractor 'tap-ibkr'...
Installing extractor 'tap-csv'...
Installed file bundle 'airflow'
Run `meltano upgrade files` to update your project's 'airflow' files.
Installed extractor 'tap-csv'
Installed loader 'target-postgres'
Installed orchestrator 'airflow'
Installed 4/5 plugins
Skipped installing 1/5 plugins
(meltano) ubuntu@juju-2dd159-155:~/meltano_proj_repo$

here's how it looks in an ubuntu LXD container.. with only Python3 installed... and I aliase'd

python=python3

in

.bashrc

```meltano --log-level=debug install [2021-09-04 133802,314] [1172|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x7f20bb37c760>@sqlite:////home/ubuntu/meltano_proj_repo/.meltano/meltano.db [2021-09-04 133802,359] [1172|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): www.meltano.com:443 [2021-09-04 133802,930] [1172|MainThread|urllib3.connectionpool] [DEBUG] https://www.meltano.com:443 "GET /discovery.yml HTTP/1.1" 200 115895 Installing 5 plugins... [2021-09-04 133803,586] [1172|MainThread|asyncio] [DEBUG] Using selector: EpollSelector Installing file bundle 'airflow'... [2021-09-04 133803,587] [1172|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'files/airflow' [2021-09-04 133803,587] [1172|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/airflow Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/airflow' [2021-09-04 133803,587] [1172|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'files/airflow' Installing orchestrator 'airflow'... [2021-09-04 133803,592] [1172|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'orchestrators/airflow' [2021-09-04 133803,593] [1172|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/airflow Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/airflow' [2021-09-04 133803,593] [1172|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'orchestrators/airflow' Installing loader 'target-postgres'... [2021-09-04 133803,597] [1172|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'loaders/target-postgres' [2021-09-04 133803,597] [1172|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/target-postgres Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/target-postgres' [2021-09-04 133803,597] [1172|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'loaders/target-postgres' Installing extractor 'tap-ibkr'... Installing extractor 'tap-csv'... [2021-09-04 133803,601] [1172|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'extractors/tap-csv' [2021-09-04 133803,601] [1172|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/tap-csv Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/tap-csv' [2021-09-04 133803,601] [1172|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'extractors/tap-csv' File bundle 'airflow' could not be installed: could not create the virtualenv for 'files/airflow' Extractor 'tap-csv' could not be installed: could not create the virtualenv for 'extractors/tap-csv' Orchestrator 'airflow' could not be installed: could not create the virtualenv for 'orchestrators/airflow' Loader 'target-postgres' could not be installed: could not create the virtualenv for 'loaders/target-postgres' Installed 0/5 plugins Skipped installing 1/5 plugins [2021-09-04 133803,67…

I even thought "oh, it just wants python instead of python3" but aliasing didn't solve it

I know 95% of users will be in docker containers but.. I prefer LXD and working with the raw install for now