https://linen.dev logo
Join Slack
Powered by
When running 'meltano install extractor tap-postgr...
# troubleshooting
m
When running 'meltano install extractor tap-postgres' I get 'failed to install plugin 'tap-postgres''. How do I see the actual error so I van have a change of maybe fixing it? With MELTANO_CLI_LOG_LEVEL=debug I am just seeing the error is raised because the install was not a succes...
w
@martijn_raats could you please post the output of running the command with the log level set to debug. It should contain the full error message from the subprocess that actually performs the install
m
martijn@Martijns-Air local-postgres-to-mysql % MELTANO_CLI_LOG_LEVEL=debug meltano add loader target-mysql ... Failed to install plugin(s) ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/__init__.py:10 │ │ 2 in _run_cli │ │ │ │ 99 │ """ │ │ 100 │ try: │ │ 101 │ │ try: # noqa: WPS225, WPS505 │ │ ❱ 102 │ │ │ cli(obj={"project": None}) │ │ 103 │ │ except ProjectReadonly as err: │ │ 104 │ │ │ raise CliError( │ │ 105 │ │ │ │ f"The requested action could not be completed: {err}", │ │ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1157 in │ │ call │ │ │ │ 1154 │ │ │ 1155 │ def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any: │ │ 1156 │ │ """Alias for meth`main`.""" │ │ ❱ 1157 │ │ return self.main(*args, **kwargs) │ │ 1158 │ │ 1159 │ │ 1160 class Command(BaseCommand): │ │ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/cli.py:42 in │ │ main │ │ │ │ 39 │ │ │ args: Positional arguments for the Click group. │ │ 40 │ │ │ kwargs: Keyword arguments for the Click group. │ │ 41 │ │ """ │ │ ❱ 42 │ │ return super().main(*args, windows_expand_args=False, **kwargs) │ │ 43 │ │ 44 │ │ 45 @click.group( │ │ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:10…
│ in invoke │ │ │ │ 609 │ │ enact_environment_behavior(self.environment_behavior, ctx) │ │ 610 │ │ if ctx.obj.get("tracker"): │ │ 611 │ │ │ ctx.obj["tracker"].add_contexts(CliContext.from_click_context(ctx)) │ │ ❱ 612 │ │ super().invoke(ctx) │ │ 613 │ │ 614 │ │ 615 class InstrumentedDefaultGroup(InstrumentedGroupMixin, DefaultGroup, DYMGroup): │ │ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1688 in │ │ invoke │ │ │ │ 1685 │ │ │ │ super().invoke(ctx) │ │ 1686 │ │ │ │ sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) │ │ 1687 │ │ │ │ with sub_ctx: │ │ ❱ 1688 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │ │ 1689 │ │ │ │ 1690 │ │ # In chain mode we create the contexts step by step, but after the │ │ 1691 │ │ # base command has been invoked. Because at that point we do not │ │ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/utils.py:661 │ │ in invoke │ │ │ │ 658 │ │ if ctx.obj.get("tracker"): │ │ 659 │ │ │ ctx.obj["tracker"].add_contexts(CliContext.from_click_context(ctx)) │ │ 660 │ │ │ ctx.obj["tracker"].track_command_event(CliEvent.started) │ │ ❱ 661 │ │ super().invoke(ctx) │ │ 662 │ │ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1434 in │ │ invoke │ │ │ │ 1431 │ │ │ echo(style(message, fg="red"), err=True) │ │ 1432 │ │ │ │ 1433 │ │ if self.callback is not None: │ │ ❱ 1434 │ │ │ return ctx.invoke(self.callback, **ctx.params) │ │…
│ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/params.py:27 │ │ in decorate │ │ │ │ 24 │ │ if database_uri: │ │ 25 │ │ │ ProjectSettingsService.config_override["database_uri"] = database_uri │ │ 26 │ │ │ │ ❱ 27 │ │ return func(*args, **kwargs) │ │ 28 │ │ │ 29 │ return functools.update_wrapper(decorate, func) │ │ 30 │ │ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/params.py:75 │ │ in decorate │ │ │ │ 72 │ │ │ │ except MigrationError as err: │ │ 73 │ │ │ │ │ raise CliError(str(err)) from err │ │ 74 │ │ │ │ │ ❱ 75 │ │ │ func(project, *args, **kwargs) │ │ 76 │ │ │ │ 77 │ │ return functools.update_wrapper(decorate, func) │ │ 78 │ │ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/decorators.py:33 in │ │ new_func │ │ │ │ 30 │ """ │ │ 31 │ │ │ 32 │ def new_func(*args: "P.args", **kwargs: "P.kwargs") -> "R": │ │ ❱ 33 │ │ return f(get_current_context(), *args, **kwargs) │ │ 34 │ │ │ 35 │ return update_wrapper(new_func, f) │ │ 36 │ │ │ │ /Users/martijn/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/add.py:163 in │ │ add │ │ │ │ 160 │ │ …
e
@martijn_raats I'm seeing
Copy code
ERROR: Package 'thk-target-mysql' requires a different Python: 3.11.5 not in '<=3.11,>=3.8.1'
So you could try using Python < 3.11. Though I'm also seeing a build error even using Python 3.10:
Copy code
Exception: Can not find valid pkg-config name.
Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually
I see the library only has Windows wheels, and I'm probably missing some dev libraries.
m
Where are you seeing that? I cannot find both messages in the log i sent you?
e
I tried installing with `pipx`:
Copy code
pipx install git+<https://github.com/thkwag/target-mysql.git> --python python3.11
m
oke thats nice to know, but my original question was how to get the sub process output when executing from meltano commands. If you don't get usefull information and if you don't know what pip command is executed in the back you can basicly not fix an issue.
e
Above the 
rich
traceback that you shared you should see normal logging where there's one or more messages like
Copy code
Installing with args 'thk-target-mysql' into virtual environment for 'loaders/target-mysql
at debug level. Which means Meltano is running 
pip install thk-target-mysql
under the hood. We could certainly surface the pip error, which I think is printed to the subprocess' stdout. If you are interested in looking at the code and maybe submitting a PR, the relevant location is here. Maybe it's as simple as adding something like
Copy code
stdout = await err.process.stdout.read()
logger.error(stdout.decode("utf-8"))
w
@edgar_ramirez_mondragon I can't check right now, but I think I updated Meltano to print out the subprocess error as part of the 
python
setting feature released in v3.0.0
e
I think pip's stderr is getting gobbled up because the 
pip install
subprocess is missing an 
extract_stderr
callback?
w
Seems so. Might be a quick fix
m
Ok, I am sorry but cannot really help because I don't know python, just wanted tot point out that this was the first command I try with meltano and without any pointers to the errors (for examle I only had to install git on my docker image) you quickly move on to other solutions.
e
Makes sense. Perhaps it's as simple at directing pip logs to a known location: https://github.com/meltano/meltano/pull/8145.
u
That PR has been merged :)
Open in Slack
PreviousNext
Powered by