Does anyone have any funky logging configs for loc...
# best-practices
d
Does anyone have any funky logging configs for local console?
đź‘€ 1
e
d
My config gets pretty verbose with all of the key-value pairs at the end:
Copy code
2024-09-27T04:52:57.806328Z [info     ] 2024-09-27 14:52:57,805 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"stream": "public-usage_events", "pid": 38540, "context": {}}} cmd_type=elb consumer=False job_name=experimental:tap-rockefeller-v3-to-target-warehouse-postgres name=tap-rockefeller-v3 producer=True run_id=2faf1b21-c9b0-4e25-98d3-5bf802bcd646 stdio=stderr string_id=tap-rockefeller-v3
2024-09-27T04:52:57.918855Z [info     ] 2024-09-27 14:52:57,918 | INFO     | target-postgres      | Target 'target-postgres' completed reading 2841 lines of input (5 schemas, 2830 records, 0 batch manifests, 6 state messages). cmd_type=elb consumer=True job_name=experimental:tap-rockefeller-v3-to-target-warehouse-postgres name=target-warehouse-postgres producer=False run_id=2faf1b21-c9b0-4e25-98d3-5bf802bcd646 stdio=stderr string_id=target-warehouse-postgres
2024-09-27T04:52:58.021565Z [info     ] 2024-09-27 14:52:58,020 | INFO     | target-postgres.public-customers | Inserting with SQL: INSERT INTO "32ac6651_f5e3_4506_a41e_584c144613e2" (customer_id, chargify_id, customer_type, email_domain, _sdc_extracted_at, _sdc_received_at, _sdc_batched_at, _sdc_deleted_at, _sdc_sequence, _sdc_table_version, _sdc_sync_started_at) VALUES (:customer_id, :chargify_id, :customer_type, :email_domain, :_sdc_extracted_at, :_sdc_received_at, :_sdc_batched_at, :_sdc_deleted_at, :_sdc_sequence, :_sdc_table_version, :_sdc_sync_started_at) cmd_type=elb consumer=True job_name=experimental:tap-rockefeller-v3-to-target-warehouse-postgres name=target-warehouse-postgres producer=False run_id=2faf1b21-c9b0-4e25-98d3-5bf802bcd646 stdio=stderr string_id=target-warehouse-postgres
Copy code
version: 1
disable_existing_loggers: false

formatters:
  structured_plain_no_locals:
    (): meltano.core.logging.console_log_formatter
    colors: False
    show_locals: False

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: structured_plain_no_locals
    stream: "<ext://sys.stderr>"

root:
  level: INFO
  propagate: yes
  handlers: [console]
e
Hey! Just to confirm: is that config good for you? It's still probably a bit on the verbose end, I imagine. We've got a feature request to make it easier to change the logging format without having to create a full log config yaml. One option could be a much simpler format without all the structured information. Do give it a 👍 or comment on it if that sounds useful. https://github.com/meltano/meltano/issues/3002
d
I'm actually getting an error from that config:
Copy code
Unable to configure handler 'file'
I think the main thing I'm after is the ability to suppress some of the key/value pairs.
These ones:
Copy code
cmd_type=elb consumer=True job_name=experimental:tap-rockefeller-v3-to-target-warehouse-postgres name=target-warehouse-postgres producer=False run_id=2faf1b21-c9b0-4e25-98d3-5bf802bcd646 stdio=stderr string_id=target-warehouse-postgres
In a json logger in production they're useful, but in a console logger for local they're noisy.
e
Gotcha. I don't if there's a way to exclude keys, or whether we can expose a simpler formatter. If you wanna investigate for yourself in the meantime. This is the logging approach Meltano uses with structlog: https://www.structlog.org/en/stable/standard-library.html#rendering-using-structlog-based-formatters-within-logging. And the logging is configured in this module: https://github.com/meltano/meltano/blob/d8d7f9fe158c8a27e7aaea130161514d4798bbf0/src/meltano/core/logging/__init__.py#L1.