Minor issue with a

meltano elt ...

job where it appears to complete fine, saves the incremental state but then throws an error, without too much detail. This is running remotely so don't have direct access to the logs

Hey @matt_arderne! If you run in debug mode (

meltano --log-level=debug elt ...

) are you able to see more detail?

Hey @edgar_ramirez_mondragon I'll have to do that, thanks!

Hi @matt_arderne, while this is not related to you issue, I noted Ascii Control characters in your log. I had this problem when I upgraded to the latest version of Meltano. If you want to get rid of the control characters and have plain ascii text, you can drop a logging.yaml file in your project like so. This is courtesy of Florian who helped me solve my log formatting issue.

version: 1
disable_existing_loggers: false

formatters:
  default:
    format: "[%(asctime)s] [%(process)d|%(threadName)10s|%(name)s] [%(levelname)s] %(message)s"
  structured_plain:
    (): meltano.core.logging.console_log_formatter
    colors: False
  structured_colored:
    (): meltano.core.logging.console_log_formatter
    colors: True
  key_value:
    (): meltano.core.logging.key_value_formatter
    sort_keys: False
  json:
    (): meltano.core.logging.json_formatter

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: structured_plain
    stream: "<ext://sys.stderr>"

root:
  level: INFO
  propagate: yes
  handlers: [console]

Hi @steve_clarke thanks for that, yeah was wondering. Do you know if there is an open issue for it?

Hi @matt_arderne, I'm not sure if there is an issue raised on this. For me, I see this logging as a great new feature - I have moved to the json logger as it will help me process the logs in Cloudwatch. If however if you are coming from an older version - you probably need to know that you need to drop in this logging.yaml file (I didn't initially). I'm not sure if the init for a new project creates a default logging.yaml file or not? It might be something worth mentioning to the Meltano team as it does affect some use cases. I think the default setting is colour logs, and in a container environment logging to something like Airflow or CloudWatch you may not want this default - structured_plain or perhaps json would be a better default.

Great thanks @steve_clarke, appreciate the insights, will have a look at the docs

I'm not sure if the init for a new project creates a default logging.yaml file or not?

Thanks for pointing this out @steve_clarke. We certainly don't include a logging config file in new projects at the moment, but that's a great idea! I created https://gitlab.com/meltano/meltano/-/issues/3121 Feel free to drop a comment in the issue if there's anything you'd like to see in the default

logging.yaml

😄 cc @taylor

@edgar_ramirez_mondragon , can we not create a ENV variable just like

MELTANO_CLI_LOG_LEVEL

. i.e :

MELTANO_CLI_LOG_FORMATTER=structured_plain

It will be much easier.

@atif_imam that'd be a interesting approach 🤔. Do file an issue describing what you'd like to see 🙂

@edgar_ramirez_mondragon https://gitlab.com/meltano/meltano/-/issues/3365 Done

I’m commenting as much in the issue, but I’m assuming that https://docs.meltano.com/reference/settings#clilog_config is a bit more complex than you were hoping for @atif_imam and you’re wanting something simpler?