Hello all. What is best practice to set up and run meltano in production, so it produces meaningful error messages? For instance, I run meltano like this, with increased logging:

meltano --log-level=debug run tap-oracle target-oracle 2>error.log

After some time it crashed, but the error message is quite impossible to understand. (I paste this message below). I then run meltano tap separately (meltano invoke tap-oracle), and then it did print the error message, allowing to identify the problem, which was badly encoded data:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 1999: unexpected end of data

However, that error was not present in the original error stream of meltano run. In production I cannot afford to re-run things, just to find errors. I need the tool to print the message in the log file. So, how do I set up meltano for production setup, so that, in the case of problems or crashes, the error messages are logged correctly? (here is the printout of the tail of the error log of meltano run - with no core error message present):

│ /home/pablo/.conda/envs/meltanop39/lib/python3.9/site-packages/meltano/cli/run.py:183 in run    │

│                                                                                                  │

│   180 │   )                                                                                      │

│   181 │                                                                                          │

│   182 │   try:                                                                                   │

│ ❱ 183 │   │   await _run_blocks(tracker, parsed_blocks, dry_run=dry_run)                         │

│   184 │   except Exception as err:                                                               │

│   185 │   │   tracker.track_command_event(CliEvent.failed)                                       │

│   186 │   │   raise err                                                                          │

│                                                                                                  │

│ /home/pablo/.conda/envs/meltanop39/lib/python3.9/site-packages/meltano/cli/run.py:230 in        │

│ _run_blocks                                                                                      │

│                                                                                                  │

│   227 │   │   │   )                                                                              │

│   228 │   │   │   with tracker.with_contexts(tracking_ctx):                                      │

│   229 │   │   │   │   tracker.track_block_event(blk_name, BlockEvents.failed)                    │

│ ❱ 230 │   │   │   raise CliError(                                                                │

│   231 │   │   │   │   f"Run invocation could not be completed as block failed: {err}",  # noqa   │

│   232 │   │   │   ) from err                                                                     │

│   233 │   │   except Exception as bare_err:                                                      │

╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

CliError: Run invocation could not be completed as block failed: Extractor failed

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮

│ /home/pablo/.conda/envs/meltanop39/lib/python3.9/site-packages/meltano/cli/__init__.py:113 in   │

│ _run_cli                                                                                         │

│                                                                                                  │

│   110 │   │   except MeltanoError as err:                                                        │

│   111 │   │   │   handle_meltano_error(err)                                                      │

│   112 │   │   except Exception as err:                                                           │

│ ❱ 113 │   │   │   raise CliError(f"{troubleshooting_message}\n{err}") from err  # noqa: EM102    │

│   114 │   except CliError as cli_error:                                                          │

│   115 │   │   cli_error.print()                                                                  │

│   116 │   │   sys.exit(cli_error.exit_code)                                                      │

╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

CliError: Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to

join our friendly Slack community.

Run invocation could not be completed as block failed: Extractor failed

Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to

join our friendly Slack community.

Run invocation could not be completed as block failed: Extractor failed

You should be able to also the extractor logs, identified with

producer=True

. Now, tbf they're not easy to find and I'm thinking of what approaches could help improve the error logging situation. • https://github.com/meltano/meltano/issues/6641 • https://github.com/meltano/meltano/issues/8555