so.. I am using Juju to instantiate and run meltan...
# troubleshooting
e
so.. I am using Juju to instantiate and run meltano.. and one of the commands I run on my project is the airflow command.. but lately suddenly I get an error
Copy code
meltano invoke airflow scheduler -D
[Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/airflow/airflow.cfg'
how do I debug this? I have a pretty new meltano project but it was working a while ago..
d
The error message about the missing file is a red herring; the real issue is that the
airflow
executable is not available where Meltano expects to find it. If you run again with
meltano --log-level=debug invoke …
, you should see additional error output to that effect. Did you run
meltano install
?
The misleading error message is a known issue, by the way: https://gitlab.com/meltano/meltano/-/issues/2573
e
yea meltano installed, let me try your verbose command now.. thanks for the heads up
```meltano --log-level=debug invoke airflow scheduler -D [2021-09-03 224943,757] [2345|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x7f1ce82f2b20>@sqlite:////home/ubuntu/meltano_proj_repo/.meltano/meltano.db [2021-09-03 224943,803] [2345|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): www.meltano.com:443 [2021-09-03 224944,375] [2345|MainThread|urllib3.connectionpool] [DEBUG] https://www.meltano.com:443 "GET /discovery.yml HTTP/1.1" 200 115895 [2021-09-03 224944,860] [2345|MainThread|root] [ERROR] The
discovery.yml
manifest received from https://www.meltano.com/discovery.yml could not be parsed. [2021-09-03 224944,860] [2345|MainThread|root] [DEBUG] [Errno 13] Permission denied: '/home/ubuntu/meltano_proj_repo/.meltano/cache/discovery.yml' [2021-09-03 224944,892] [2345|MainThread|root] [DEBUG] Invoking: ['/home/ubuntu/meltano_proj_repo/.meltano/orchestrators/airflow/venv/bin/airflow', '--help'] [2021-09-03 224944,892] [2345|MainThread|root] [DEBUG] Env: {'SHELL': '/bin/bash', 'LC_ADDRESS': 'C.UTF-8', 'LC_NAME': 'C.UTF-8', 'LC_MONETARY': 'C.UTF-8', 'PWD': '/home/ubuntu/meltano_proj_repo', 'LOGNAME': 'ubuntu', 'XDG_SESSION_TYPE': 'tty', 'MOTD_SHOWN': 'pam', 'HOME': '/home/ubuntu', 'LC_PAPER': 'C.UTF-8', 'LANG': 'C.UTF-8', 'LS_COLORS': 'rs=0di=01;34ln=01;36mh=00pi=40;33so=01;35do=01;35bd=40;33;01cd=40;33;01or=40;31;01mi=00su=37;41sg=30;43ca=30;41tw=30;42ow=34;42st=37;44ex=01;32.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:*.xspf=00;36:', 'SSH_CONNECTION': '10.22.143.1 45576 10.22.143.76 22', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'TERM': 'xterm-256color', 'LC_IDENTIFICATION': 'C.UTF-8', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'USER': 'ubuntu', 'SHLVL': '1', 'LC_TELEPHONE': 'C.UTF-8', 'LC_MEASUREMENT': 'C.UTF-8', 'XDG_SESSION_ID': '415', 'XDG_RUNTIME_DIR': '/run/user/1000', 'SSH_CLIENT': '10.22.143.1 45576 22', 'LC_TIME': 'C.UTF-8', 'XDG_DATA_DIRS': '/usr/local/share/usr/share/var/lib/snapd/desktop', 'PATH': '/home/ubuntu/meltano_proj_repo/.meltano/orchestrators/airflow/venv/bin/usr/local/sbin/usr/local/bin/usr/sbin/usr/bin/sbin/bin/usr/games/usr/local/games:/snap/bin', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'SSH_TTY': '/dev/pts/0', 'LC_NUMERIC': 'C.UTF-8', 'BASH_FUNC_juju_machine_lock%%': '() { for agent in $(ls /var/lib/juju/agents);\n do\n juju_agent machinelock $agent 2> /dev/null;\n done\n}', 'BASH_FUNC_juju_statetracker_report%%': '() { juju_agent debug/pprof/juju/state/tracker?debug=1\n}', 'BASH_FUNC_juju_agent_call%%': '() { local agent=$1;\n shift;\n juju-introspect --ag…
let me double check if my juju charm does
Copy code
meltano add orchestrator airflow
thank you for the help finding the verbose mode @douwe_maan, it turned out I needed to not just add airflow but debug my tap.. which was not present in the environment. I use Juju to instantiate meltano in a container so.. the container needed to be setup properly Thanks again!
hmmm on second check.. when I start it's still failing.. let me re-read here your post and see if I missed something
ahhh i see my problem
Copy code
ubuntu@juju-2dd159-150:~/meltano_proj_repo$ meltano install 
[2021-09-04 11:26:56,249] [1192|MainThread|root] [ERROR] The `discovery.yml` manifest received from https://www.meltano.com/discovery.yml could not be parsed.
Extractor 'tap-ibkr' is not known to Meltano
trying to patch this up but.. seems to fail to create the virtualenv
```meltano --log-level=debug add --custom extractor tap-ibkr [2021-09-04 114113,125] [1384|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x7f07901fddf0>@sqlite:////home/ubuntu/meltano_proj_repo/.meltano/meltano.db Adding new custom extractor with name 'tap-ibkr'... Specify the plugin's namespace, which will serve as the: - identifier to find related/compatible plugins - default database schema (
load_schema
extra), for use by loaders that support a target schema Hit Return to accept the default: plugin name with underscores instead of dashes (namespace) [tap_ibkr]: Specify the plugin's
pip install
argument, for example: - PyPI package name: tap-ibkr - Git repository URL: git+https://gitlab.com/meltano/tap-ibkr.git - local directory, in editable/development mode: -e extract/tap-ibkr - 'n' if using a local executable (nothing to install) Default: plugin name as PyPI package name (pip_url) [tap-ibkr]: /home/ubuntu/meltano_tap_ibkr/tap-ibkr Specify the plugin's executable name Default: name derived from
pip_url
(executable) [tap-ibkr]: Specify the tap's supported Singer features (executable flags), for example: `catalog`: supports the
--catalog
flag `discover`: supports the
--discover
flag `properties`: supports the
--properties
flag `state`: supports the
--state
flag To find out what features a tap supports, reference its documentation or try one of the tricks under https://meltano.com/docs/contributor-guide.html#how-to-test-a-tap. Multiple capabilities can be separated using commas. Default: no capabilities (capabilities) [[]]: Specify the tap's supported settings (
config.json
keys) Multiple setting names (keys) can be separated using commas. A setting kind can be specified alongside the name (key) by using the
:
delimiter, e.g.
port:integer
to set the kind
integer
for the name
port
Supported setting kinds: string | integer | boolean | date_iso8601 | email | password | oauth | options | file | array | object | hidden - Credentials and other sensitive setting types should use the password kind. - If not specified, setting kind defaults to string. - Nested properties can be represented using the
.
separator, e.g.
auth.username
for
{ "auth": { "username": value } }
. - To find out what settings a tap supports, reference its documentation. Default: no settings (settings) [[]]: [2021-09-04 114149,254] [1384|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): www.meltano.com:443 [2021-09-04 114149,826] [1384|MainThread|urllib3.connectionpool] [DEBUG] https://www.meltano.com:443 "GET /discovery.yml HTTP/1.1" 200 115895 [2021-09-04 114150,324] [1384|MainThread|root] [ERROR] The
discovery.yml
manifest received from https://www.meltano.com/discovery.yml could not be parsed. [2021-09-04 114150,324] [1384|MainThread|root] [DEBUG] [Errno 13] Permission denied: '/home/ubuntu/meltano_proj_repo/.meltano/cache/discovery.yml' Added extractor 'tap-ibkr' to your Meltano project [2021-09-04 114150,368] [1384|MainThread|asyncio] [DEBUG] Using selector: EpollSelector Installing extractor 'tap-ibkr'... [2021-09-04 114150,369] [1384|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'extractors/tap-ibkr' [2021-09-04 114150,369] [1384|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/tap-ibkr Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/tap-ibkr' [2021-09-04 114150,369] [1384|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'extractors/tap-ibkr' Extractor 'tap-ibkr' could not be installed: could not create the virtua…
im quite stuck now.. but basically I want to simply run a private tap so I can develop it and improve it..
Copy code
- name: tap-ibkr
    pip_url: git+ssh://git@github.com/MYGROUP/REPO.git
    executable: /home/ubuntu/meltano_tap_repo/tap-ibkr/tap-ibkr.sh
but this doesn't seem to be enough and then I goto try re-adding or re-installing it in the yaml but getting stuck there with what seems like permission issues?
it's a private repo too.. maybe thats the issue?
the code is already cloned but.. unsure maybe it's unhappy with somethign because it's a SSH git url?
ahhh finally I figured it all out
turns out... Meltano doesn't install without being used in a virtualized python .. EVEN in a containerized environment.. it just fails fails fails.. BUT as soon as I added a basic venv.. inside my container.. all is smooth and going well
Copy code
ubuntu@juju-2dd159-155:~/meltano_proj_repo$ python -m venv .venv/meltano
ubuntu@juju-2dd159-155:~/meltano_proj_repo$ source .venv/meltano/bin/activate
(meltano) ubuntu@juju-2dd159-155:~/meltano_proj_repo$ meltano install
Installing 5 plugins...
Installing file bundle 'airflow'...
Installing orchestrator 'airflow'...
Installing loader 'target-postgres'...
Installing extractor 'tap-ibkr'...
Installing extractor 'tap-csv'...
Installed file bundle 'airflow'
Run `meltano upgrade files` to update your project's 'airflow' files.
Installed extractor 'tap-csv'
Installed loader 'target-postgres'
Installed orchestrator 'airflow'
Installed 4/5 plugins
Skipped installing 1/5 plugins
(meltano) ubuntu@juju-2dd159-155:~/meltano_proj_repo$
here's how it looks in an ubuntu LXD container.. with only Python3 installed... and I aliase'd
python=python3
in
.bashrc
```meltano --log-level=debug install [2021-09-04 133802,314] [1172|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x7f20bb37c760>@sqlite:////home/ubuntu/meltano_proj_repo/.meltano/meltano.db [2021-09-04 133802,359] [1172|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): www.meltano.com:443 [2021-09-04 133802,930] [1172|MainThread|urllib3.connectionpool] [DEBUG] https://www.meltano.com:443 "GET /discovery.yml HTTP/1.1" 200 115895 Installing 5 plugins... [2021-09-04 133803,586] [1172|MainThread|asyncio] [DEBUG] Using selector: EpollSelector Installing file bundle 'airflow'... [2021-09-04 133803,587] [1172|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'files/airflow' [2021-09-04 133803,587] [1172|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/airflow Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/airflow' [2021-09-04 133803,587] [1172|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'files/airflow' Installing orchestrator 'airflow'... [2021-09-04 133803,592] [1172|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'orchestrators/airflow' [2021-09-04 133803,593] [1172|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/airflow Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/airflow' [2021-09-04 133803,593] [1172|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'orchestrators/airflow' Installing loader 'target-postgres'... [2021-09-04 133803,597] [1172|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'loaders/target-postgres' [2021-09-04 133803,597] [1172|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/target-postgres Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/target-postgres' [2021-09-04 133803,597] [1172|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'loaders/target-postgres' Installing extractor 'tap-ibkr'... Installing extractor 'tap-csv'... [2021-09-04 133803,601] [1172|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'extractors/tap-csv' [2021-09-04 133803,601] [1172|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/tap-csv Traceback (most recent call last): File "/usr/lib/python3.8/shutil.py", line 707, in rmtree orig_st = os.lstat(path) FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/tap-csv' [2021-09-04 133803,601] [1172|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'extractors/tap-csv' File bundle 'airflow' could not be installed: could not create the virtualenv for 'files/airflow' Extractor 'tap-csv' could not be installed: could not create the virtualenv for 'extractors/tap-csv' Orchestrator 'airflow' could not be installed: could not create the virtualenv for 'orchestrators/airflow' Loader 'target-postgres' could not be installed: could not create the virtualenv for 'loaders/target-postgres' Installed 0/5 plugins Skipped installing 1/5 plugins [2021-09-04 133803,67…
d
@emcp Great job debugging! That venv requirement is definitely a bug. Can you please create a bug report issue?
e
can do.. yeah i was going nuts thinking... come on what is up with this
I even thought "oh, it just wants python instead of python3" but aliasing didn't solve it
I know 95% of users will be in docker containers but.. I prefer LXD and working with the raw install for now
d
Totally reasonable, and something we should support :) thanks for working this out so the next person won’t need to