Hey everyone, I believe since early 2022, the default logging has changed and we should now use a

logging.yaml

file. This works well but there’s still something we’d like to change because it makes it hard to read logs. After each log line, we get some extra values, e.g.

cmd_type=loader job_id=tap-salesforce-sfdc-period-target-presto-parquet name=target-presto-parquet run_id=87924270-556e-483e-ae97-5ff9f6fd2ab7 stdio=stderr

Here’s our

logging.yaml

file:

version: 1
disable_existing_loggers: false

formatters:
  default:
    format: "[%(asctime)s] [%(process)d|%(threadName)10s|%(name)s] [%(levelname)s] %(message)s"
  structured_plain:
    (): meltano.core.logging.console_log_formatter
    colors: False
  structured_colored:
    (): meltano.core.logging.console_log_formatter
    colors: True
  key_value:
    (): meltano.core.logging.key_value_formatter
    sort_keys: False
  json:
    (): meltano.core.logging.json_formatter

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: structured_plain
    stream: "<ext://sys.stderr>"
  file:
    class: logging.FileHandler
    level: INFO
    filename: /var/log/meltano.log
    formatter: json

root:
  level: INFO
  propagate: yes
  handlers: [console, file]

and here’s a sample log file:

INFO Used 1582855 of 39932200 daily REST API quota cmd_type=extractor job_id=tap-salesforce-sfdc-period-target-presto-parquet name=tap-salesforce run_id=87924270-556e-483e-ae97-5ff9f6fd2ab7 stdio=stderr
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 408, "tags": {"endpoint": "Period"}} cmd_type=extractor job_id=tap-salesforce-sfdc-period-target-presto-parquet name=tap-salesforce run_id=87924270-556e-483e-ae97-5ff9f6fd2ab7 stdio=stderr
INFO Setting state to {'bookmarks': {'Period': {'version': 1651764951849, 'SystemModstamp': '2022-05-04T12:46:22.000000Z'}}} cmd_type=loader job_id=tap-salesforce-sfdc-period-target-presto-parquet name=target-presto-parquet run_id=87924270-556e-483e-ae97-5ff9f6fd2ab7 stdio=stderr
INFO Completed sync for Period cmd_type=extractor job_id=tap-salesforce-sfdc-period-target-presto-parquet name=tap-salesforce run_id=87924270-556e-483e-ae97-5ff9f6fd2ab7 stdio=stderr
INFO Finished sync             cmd_type=extractor job_id=tap-salesforce-sfdc-period-target-presto-parquet name=tap-salesforce run_id=87924270-556e-483e-ae97-5ff9f6fd2ab7 stdio=stderr

we would like to remove all the extra

cmd_type=extractor job_id=tap-salesforce-sfdc-period-target-presto-parquet name=tap-salesforce run_id=87924270-556e-483e-ae97-5ff9f6fd2ab7 stdio=stderr

and have the log file as below instead

INFO Used 1582855 of 39932200 daily REST API quota
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 408, "tags": {"endpoint": "Period"}}
INFO Setting state to {'bookmarks': {'Period': {'version': 1651764951849, 'SystemModstamp': '2022-05-04T12:46:22.000000Z'}}}
INFO Completed sync
INFO Finished sync

Does anyone have any idea how to make that happen? I’ve tried to update the

logging.yaml

file in many different ways but without success… Thanks a mil!

That's an interesting one... The formatters are defined in the meltano source core.logging.formatters, and pulls from https://www.structlog.org/en/stable/. I don't see where meltano would read a user-defined Python file to define additional formatters, to allow them to be selected via the logging.yaml file. However, logging.yaml is processed by the standard logging module, I believe, so you can try throwing your own format in there, as it shows with

default

. Set your formatter to default under handlers.console, then play around with the format string...

Yep so I tried all this but basically the

message

object contains a bunch of keys, from which I only want one:

event

. Unfortunately, I'm not able to extract only this one

event

key because it's a dict formatted as a string…

Looking at the code, it looks like it is "hard coded" to use structured logs with those specific keys and values, so it does not look like this is currently something that can be overridden.

Yep that's what I figured as well. Thanks for looking into it too! Appreciate it :)

Hey folks, I have a similar use case, and since its been 8 months , i am tempted to ask , was this ever handled? 🙂