https://linen.dev logo
Join Slack
Powered by
Hi all! I currently have a pipeline working in a l...
# troubleshooting
c
Hi all! I currently have a pipeline working in a local installation and I wanted to containerise the project, however I am having trouble with discovery on tap-oracle. When I run the docker image I get the following,
Copy code
ELT could not be completed: Cannot start extractor: Catalog discovery failed
invoke tap-oracle -d
 also does not work. Discovery for tap-oracle on local installation works fine but once I use docker, it does not. Anyone else had trouble with containerising tap-oracle?
v
You'll need to get the actual error that's happening, so you're probably running 
meltano elt tap-oracle target-jsonl
? if so then add 
meltano --log-level=debug elt tap-oracle target-jsonl
I run tap-oracle in production, I don't see any reason this shouldn't work, maybe there's a dependency missing or something
c
The log is relatively long so here's a snippet when `meltano --log-level=debug elt tap-oracle target-postgres`is run within the docker image
Copy code
2022-01-07T13:53:35.930869Z [debug    ] Deleted configuration at /project/.meltano/run/elt/2022-01-07T135334--tap-oracle--target-postgres/f6df6546-1863-4085-bc61-7ff0cd26999d/target.7ab83b2e-e976-4e6e-90f6-15790c1a5ac3.config.json
2022-01-07T13:53:35.931887Z [debug    ] Deleted configuration at /project/.meltano/run/elt/2022-01-07T135334--tap-oracle--target-postgres/f6df6546-1863-4085-bc61-7ff0cd26999d/tap.5718d68d-105d-4e88-ae8d-60abaf1a77d8.config.json
2022-01-07T13:53:35.933225Z [debug    ] ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/project/.meltano/extractors/tap-oracle/venv/bin/tap-oracle', '--config', '/project/.meltano/run/elt/2022-01-07T135334--tap-oracle--target-postgres/f6df6546-1863-4085-bc61-7ff0cd26999d/tap.5718d68d-105d-4e88-ae8d-60abaf1a77d8.config.json', '--discover'] returned 1 
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 67, in invoke
    stderr=asyncio.subprocess.PIPE,  # Log
  File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 277, in invoke_async
    async with self._invoke(*args, **kwargs) as (
  File "/usr/local/lib/python3.6/site-packages/async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin_invoker.py", line 262, in _invoke
    async with self.plugin.trigger_hooks("invoke", self, args):
  File "/usr/local/lib/python3.6/site-packages/async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 87, in trigger_hooks
    await self.__class__.trigger(self, f"before_{hook_name}", *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 115, in trigger
    raise err
  File "/usr/local/lib/python3.6/site-packages/meltano/core/behavior/hookable.py", line 107, in trigger
    await hook_func(target, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/tap.py", line 270, in discover_catalog_hook
    await self.discover_catalog(plugin_invoker)
  File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/tap.py", line 315, in discover_catalog
    await self.run_discovery(plugin_invoker, catalog_path)
  File "/usr/local/lib/python3.6/site-packages/meltano/core/plugin/singer/tap.py", line 382, in run_discovery
    f"Catalog discovery failed: command {plugin_invoker.exec_args('--discover')} returned {exit_code}"
meltano.core.plugin.error.PluginExecutionError: Catalog discovery failed: command ['/project/.meltano/extractors/tap-oracle/venv/bin/tap-oracle', '--config', '/project/.meltano/run/elt/2022-01-07T135334--tap-oracle--target-postgres/f6df6546-1863-4085-bc61-7ff0cd26999d/tap.5718d68d-105d-4e88-ae8d-60abaf1a77d8.config.json', '--discover'] returned 1
v
There's a stack trace above this one that shows your actual issue
THis one is Meltano's stack trace, above will be your tap's stack trace
Also Python 3.6 is no longer supported as of December 23 2021
@amanda.folson can you update this notice to go off on python3.6 as well? 😄
c
Here's the stack trace that's above, it includes credentials but this is only a PoC env anyway ```2022-01-07T135334.508923Z [debug ] Project is read-only. 2022-01-07T135334.598366Z [debug ] Creating engine <meltano.core.project.Project object at 0x7f0a475e6cf8>@sqlite:////project/.meltano/meltano.db 2022-01-07T135334.659957Z [debug ] Using selector: EpollSelector 2022-01-07T135335.469661Z [info ] Running extract & load... job_id=2022-01-07T135334--tap-oracle--target-postgres name=meltano run_id=f6df6546-1863-4085-bc61-7ff0cd26999d 2022-01-07T135335.526667Z [debug ] Created configuration at /project/.meltano/run/elt/2022-01-07T135334--tap-oracle--target-postgres/f6df6546-1863-4085-bc61-7ff0cd26999d/tap.5718d68d-105d-4e88-ae8d-60abaf1a77d8.config.json 2022-01-07T135335.528015Z [debug ] Could not find tap.properties.json in /project/.meltano/extractors/tap-oracle/tap.properties.json, skipping. 2022-01-07T135335.529449Z [debug ] Could not find tap.properties.cache_key in /project/.meltano/extractors/tap-oracle/tap.properties.cache_key, skipping. 2022-01-07T135335.530404Z [debug ] Could not find state.json in /project/.meltano/extractors/tap-oracle/state.json, skipping. 2022-01-07T135335.586840Z [debug ] Created configuration at /project/.meltano/run/elt/2022-01-07T135334--tap-oracle--target-postgres/f6df6546-1863-4085-bc61-7ff0cd26999d/target.7ab83b2e-e976-4e6e-90f6-15790c1a5ac3.config.json 2022-01-07T135335.589775Z [debug ] Invoking: ['/project/.meltano/extractors/tap-oracle/venv/bin/tap-oracle', '--config', '/project/.meltano/run/elt/2022-01-07T135334--tap-oracle--target-postgres/f6df6546-1863-4085-bc61-7ff0cd26999d/tap.5718d68d-105d-4e88-ae8d-60abaf1a77d8.config.json', '--discover'] 2022-01-07T135335.590667Z [debug ] Env: {'PATH': '/project/.meltano/extractors/tap-oracle/venv/bin/usr/local/bin/usr/local/sbin/usr/local/bin/usr/sbin/usr/bin/sbin:/bin', 'HOSTNAME': '58e7e6691ed0', 'LANG': 'C.UTF-8', 'GPG_KEY': '0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D', 'PYTHON_VERSION': '3.6.15', 'PYTHON_PIP_VERSION': '21.2.4', 'PYTHON_SETUPTOOLS_VERSION': '57.5.0', 'PYTHON_GET_PIP_URL': 'https://github.com/pypa/get-pip/raw/3cb8888cc2869620f57d5d2da64da38f516078c7/public/get-pip.py', 'PYTHON_GET_PIP_SHA256': 'c518250e91a70d7b20cceb15272209a4ded2a0c263ae5776f129e0d9b5674309', 'NODE_VERSION': '10', 'MELTANO_PROJECT_READONLY': 'true', 'HOME': '/root', 'MELTANO_JOB_TRIGGER': 'cli', 'MELTANO_PROJECT_ROOT': '/project', 'MELTANO_SEND_ANONYMOUS_USAGE_STATS': 'true', 'MELTANO_PROJECT_ID': '5d99e12f-73c2-4678-88aa-34690713c0f7', 'MELTANO_DATABASE_URI': 'sqlite:////project/.meltano/meltano.db', 'MELTANO_DATABASE_MAX_RETRIES': '3', 'MELTANO_DATABASE_RETRY_TIMEOUT': '5', 'MELTANO_DISCOVERY_URL': 'https://www.meltano.com/discovery.yml', 'MELTANO_ELT_BUFFER_SIZE': '10485760', 'MELTANO_CLI_LOG_LEVEL': 'debug', 'MELTANO_CLI_LOG_CONFIG': 'logging.yaml', 'MELTANO_UI_BIND_HOST': '0.0.0.0', 'MELTANO_API_HOSTNAME': '0.0.0.0', 'MELTANO_UI_BIND_PORT': '5000', 'MELTANO_API_PORT': '5000', 'PORT': '5000', 'MELTANO_UI_SESSION_COOKIE_SECURE': 'false', 'MELTANO_UI_SECRET_KEY': 'thisisnotapropersecretkey', 'MELTANO_UI_PASSWORD_SALT': 'b4c124932584ad6e69f2774a0ae5c138', 'MELTANO_UI_WORKERS': '4', 'WORKERS': '4', 'WEB_CONCURRENCY': '4', 'MELTANO_UI_FORWARDED_ALLOW_IPS': '127.0.0.1', 'FORWARDED_ALLOW_IPS': '127.0.0.1', 'MELTANO_UI_READONLY': 'false', 'MELTANO_READONLY': 'false', 'MELTANO_UI_AUTHENTICATION': 'false', 'MELTANO_AUTHENTICATION': 'false', 'MELTANO_UI_ANONYMOUS_READONLY': 'false', 'MELTANO_UI_NOTIFICATION': 'false', 'MELTANO_NOTIFICATION': 'false', 'MELTANO_UI_ANALYSIS': 'false', 'MAIL_SERVER': 'localhost', 'MELTANO_MAIL_SERVER': 'localhost', 'MAIL_PORT': '1025', 'MELTANO_MAIL_PORT': '1025', 'MAIL_DEFAULT_SENDER': '"Meltano" <bot@meltano.com>', 'MELTANO_MAIL_DEFAULT_SENDER': '"Meltano" <bot@meltano.com>', 'MAIL_USE_TLS': 'false', 'MELTANO_MAIL_USE_TLS': 'false', 'MAIL_DEBUG': '…
I also have Python 3.8.10 installed on my server
v
Do you see the issue in there? 😄
Copy code
2022-01-07T13:53:35.907479Z [info     ] cx_Oracle.DatabaseError: DPI-1047: 64-bit Oracle Client library cannot be loaded: "libclntsh.so: cannot open shared object file: No such file or directory". See <https://oracle.github.io/odpi/doc/installation.html#linux> for help name=tap-oracle stdio=stderr type=discovery
https://github.com/transferwise/pipelinewise-tap-oracle/issues/7 😄 I feel like you and I were at the same point in life
c
I thought that error may have been the issue but my experience with Oracle is very limited so wasn't able to comment on that
What is the actual meaning of the error?
I previously installed relevant instantclient packages to connect Oracle outside of Meltano and the connection works fine. It seems there's a connectivity issue within the docker image (if I'm understanding the error correctly). I'm certainly missing something 🤔
These are the Oracle packages currently installed
Copy code
oracle-instantclient-basic         21.4.0.0.0-2                          amd64        Oracle Instant Client Basic package
oracle-instantclient-devel         21.4.0.0.0-2                          amd64        Development header files for Oracle Instant Client.
oracle-instantclient-sqlplus       21.4.0.0.0-2                          amd64        Oracle Instant Client SQL*Plus package
oracle-instantclient-tools         21.4.0.0.0-2                          amd64        Tools for Oracle Instant Client
v
The meaning is that it can't find the oracle instant client, did you follow the install steps from oracles side?
c
Ahh I've solved the issue. I hadn't yet installed oracle instant client on the docker image itself (only on the server it runs on). There was no reference to Oracle within the Dockerfile so of course it wouldn't connect. Thanks @visch 👍
v
Glad to help, welcome to the Meltano party 😄
one of us
Open in Slack
PreviousNext
Powered by