florian_ernst
03/14/2024, 4:56 PMrun_id
to JSON logs
When using a JSON formatter, the run_id
isn’t specified in the resulting logs. Here’s an example line in meltano.log
⤵️
{
"consumer": true,
"producer": false,
"string_id": "target-snowflake",
"cmd_type": "elb",
"stdio": "stderr",
"name": "target-snowflake",
"event": "time=2024-03-14 16:41:30 name=target_snowflake level=INFO message=Loading into REDACTED.\"REDACTED\": {\"inserts\": 315600, \"updates\": 0, \"size_bytes\": 13682350}",
"level": "info",
"timestamp": "2024-03-14T16:41:30.459033Z"
}
Therefore, when exporting to Datadog, I can’t distinguish my different jobs. Is there a way to fix that?
I put my logging.yaml
file in the thread 🧵florian_ernst
03/14/2024, 4:56 PM# logging.yaml
version: 1
disable_existing_loggers: false
formatters:
structured_plain:
(): meltano.core.logging.console_log_formatter
colors: True
json:
(): meltano.core.logging.json_formatter
handlers:
console:
class: logging.StreamHandler
level: WARNING
formatter: structured_plain
stream: "<ext://sys.stderr>"
file:
class: logging.FileHandler
level: INFO
filename: meltano.log
formatter: json
root:
level: INFO
propagate: yes
handlers: [console, file]
Edgar Ramírez (Arch.dev)
03/14/2024, 6:24 PMrun_id
property to the logger in https://github.com/meltano/meltano/blob/74d826f3ccbe03ccc986fafa229dec7cdcaab782/src/meltano/core/block/extract_load.py#L607-L612. PRs welcome!Edgar Ramírez (Arch.dev)
03/14/2024, 8:20 PM