emcp
09/03/2021, 10:45 PMmeltano invoke airflow scheduler -D
[Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/airflow/airflow.cfg'
how do I debug this? I have a pretty new meltano project but it was working a while ago..douwe_maan
09/03/2021, 10:47 PMairflow executable is not available where Meltano expects to find it. If you run again with meltano --log-level=debug invoke …, you should see additional error output to that effect.
Did you run meltano install?douwe_maan
09/03/2021, 10:47 PMemcp
09/03/2021, 10:49 PMemcp
09/04/2021, 9:32 AMdiscovery.yml manifest received from https://www.meltano.com/discovery.yml could not be parsed.
[2021-09-03 224944,860] [2345|MainThread|root] [DEBUG] [Errno 13] Permission denied: '/home/ubuntu/meltano_proj_repo/.meltano/cache/discovery.yml'
[2021-09-03 224944,892] [2345|MainThread|root] [DEBUG] Invoking: ['/home/ubuntu/meltano_proj_repo/.meltano/orchestrators/airflow/venv/bin/airflow', '--help']
[2021-09-03 224944,892] [2345|MainThread|root] [DEBUG] Env: {'SHELL': '/bin/bash', 'LC_ADDRESS': 'C.UTF-8', 'LC_NAME': 'C.UTF-8', 'LC_MONETARY': 'C.UTF-8', 'PWD': '/home/ubuntu/meltano_proj_repo', 'LOGNAME': 'ubuntu', 'XDG_SESSION_TYPE': 'tty', 'MOTD_SHOWN': 'pam', 'HOME': '/home/ubuntu', 'LC_PAPER': 'C.UTF-8', 'LANG': 'C.UTF-8', 'LS_COLORS': 'rs=0di=01;34ln=01;36mh=00pi=40;33so=01;35do=01;35bd=40;33;01cd=40;33;01or=40;31;01mi=00su=37;41sg=30;43ca=30;41tw=30;42ow=34;42st=37;44ex=01;32.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:*.xspf=00;36:', 'SSH_CONNECTION': '10.22.143.1 45576 10.22.143.76 22', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'TERM': 'xterm-256color', 'LC_IDENTIFICATION': 'C.UTF-8', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'USER': 'ubuntu', 'SHLVL': '1', 'LC_TELEPHONE': 'C.UTF-8', 'LC_MEASUREMENT': 'C.UTF-8', 'XDG_SESSION_ID': '415', 'XDG_RUNTIME_DIR': '/run/user/1000', 'SSH_CLIENT': '10.22.143.1 45576 22', 'LC_TIME': 'C.UTF-8', 'XDG_DATA_DIRS': '/usr/local/share/usr/share/var/lib/snapd/desktop', 'PATH': '/home/ubuntu/meltano_proj_repo/.meltano/orchestrators/airflow/venv/bin/usr/local/sbin/usr/local/bin/usr/sbin/usr/bin/sbin/bin/usr/games/usr/local/games:/snap/bin', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'SSH_TTY': '/dev/pts/0', 'LC_NUMERIC': 'C.UTF-8', 'BASH_FUNC_juju_machine_lock%%': '() { for agent in $(ls /var/lib/juju/agents);\n do\n juju_agent machinelock $agent 2> /dev/null;\n done\n}', 'BASH_FUNC_juju_statetracker_report%%': '() { juju_agent debug/pprof/juju/state/tracker?debug=1\n}', 'BASH_FUNC_juju_agent_call%%': '() { local agent=$1;\n shift;\n juju-introspect --ag…emcp
09/04/2021, 9:34 AMmeltano add orchestrator airflowemcp
09/04/2021, 9:52 AMemcp
09/04/2021, 11:23 AMemcp
09/04/2021, 11:27 AMubuntu@juju-2dd159-150:~/meltano_proj_repo$ meltano install
[2021-09-04 11:26:56,249] [1192|MainThread|root] [ERROR] The `discovery.yml` manifest received from https://www.meltano.com/discovery.yml could not be parsed.
Extractor 'tap-ibkr' is not known to Meltanoemcp
09/04/2021, 11:42 AMemcp
09/04/2021, 11:42 AMload_schema extra),
for use by loaders that support a target schema
Hit Return to accept the default: plugin name with underscores instead of dashes
(namespace) [tap_ibkr]:
Specify the plugin's pip install argument, for example:
- PyPI package name:
tap-ibkr
- Git repository URL:
git+https://gitlab.com/meltano/tap-ibkr.git
- local directory, in editable/development mode:
-e extract/tap-ibkr
- 'n' if using a local executable (nothing to install)
Default: plugin name as PyPI package name
(pip_url) [tap-ibkr]: /home/ubuntu/meltano_tap_ibkr/tap-ibkr
Specify the plugin's executable name
Default: name derived from pip_url
(executable) [tap-ibkr]:
Specify the tap's supported Singer features (executable flags), for example:
`catalog`: supports the --catalog flag
`discover`: supports the --discover flag
`properties`: supports the --properties flag
`state`: supports the --state flag
To find out what features a tap supports, reference its documentation or try one
of the tricks under https://meltano.com/docs/contributor-guide.html#how-to-test-a-tap.
Multiple capabilities can be separated using commas.
Default: no capabilities
(capabilities) [[]]:
Specify the tap's supported settings (config.json keys)
Multiple setting names (keys) can be separated using commas.
A setting kind can be specified alongside the name (key) by using the : delimiter,
e.g. port:integer to set the kind integer for the name port
Supported setting kinds:
string | integer | boolean | date_iso8601 | email | password | oauth | options | file | array | object | hidden
- Credentials and other sensitive setting types should use the password kind.
- If not specified, setting kind defaults to string.
- Nested properties can be represented using the . separator, e.g. auth.username for { "auth": { "username": value } }.
- To find out what settings a tap supports, reference its documentation.
Default: no settings
(settings) [[]]:
[2021-09-04 114149,254] [1384|MainThread|urllib3.connectionpool] [DEBUG] Starting new HTTPS connection (1): www.meltano.com:443
[2021-09-04 114149,826] [1384|MainThread|urllib3.connectionpool] [DEBUG] https://www.meltano.com:443 "GET /discovery.yml HTTP/1.1" 200 115895
[2021-09-04 114150,324] [1384|MainThread|root] [ERROR] The discovery.yml manifest received from https://www.meltano.com/discovery.yml could not be parsed.
[2021-09-04 114150,324] [1384|MainThread|root] [DEBUG] [Errno 13] Permission denied: '/home/ubuntu/meltano_proj_repo/.meltano/cache/discovery.yml'
Added extractor 'tap-ibkr' to your Meltano project
[2021-09-04 114150,368] [1384|MainThread|asyncio] [DEBUG] Using selector: EpollSelector
Installing extractor 'tap-ibkr'...
[2021-09-04 114150,369] [1384|MainThread|meltano.core.venv_service] [DEBUG] Removed old virtual environment for 'extractors/tap-ibkr'
[2021-09-04 114150,369] [1384|MainThread|meltano.core.venv_service] [DEBUG] Failed to remove directory tree /home/ubuntu/meltano_proj_repo/.meltano/run/tap-ibkr
Traceback (most recent call last):
File "/usr/lib/python3.8/shutil.py", line 707, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/meltano_proj_repo/.meltano/run/tap-ibkr'
[2021-09-04 114150,369] [1384|MainThread|meltano.core.venv_service] [DEBUG] Creating virtual environment for 'extractors/tap-ibkr'
Extractor 'tap-ibkr' could not be installed: could not create the virtua…emcp
09/04/2021, 11:45 AM- name: tap-ibkr
pip_url: git+ssh://git@github.com/MYGROUP/REPO.git
executable: /home/ubuntu/meltano_tap_repo/tap-ibkr/tap-ibkr.shemcp
09/04/2021, 11:45 AMemcp
09/04/2021, 11:46 AMemcp
09/04/2021, 11:47 AMemcp
09/04/2021, 1:55 PMemcp
09/04/2021, 1:56 PMubuntu@juju-2dd159-155:~/meltano_proj_repo$ python -m venv .venv/meltano
ubuntu@juju-2dd159-155:~/meltano_proj_repo$ source .venv/meltano/bin/activate
(meltano) ubuntu@juju-2dd159-155:~/meltano_proj_repo$ meltano install
Installing 5 plugins...
Installing file bundle 'airflow'...
Installing orchestrator 'airflow'...
Installing loader 'target-postgres'...
Installing extractor 'tap-ibkr'...
Installing extractor 'tap-csv'...
Installed file bundle 'airflow'
Run `meltano upgrade files` to update your project's 'airflow' files.
Installed extractor 'tap-csv'
Installed loader 'target-postgres'
Installed orchestrator 'airflow'
Installed 4/5 plugins
Skipped installing 1/5 plugins
(meltano) ubuntu@juju-2dd159-155:~/meltano_proj_repo$emcp
09/04/2021, 1:57 PMpython=python3 in .bashrcemcp
09/04/2021, 1:57 PMdouwe_maan
09/04/2021, 2:34 PMemcp
09/04/2021, 2:35 PMemcp
09/04/2021, 2:35 PMemcp
09/04/2021, 2:35 PMdouwe_maan
09/04/2021, 2:41 PMemcp
09/04/2021, 2:43 PM