GitHub
06/19/2025, 5:42 PMrun_id to Meltano logging
Issue created by florian-ernst-alan
### Feature scope
CLI (options, error messages, logging, etc.)
### Description
### The problem
When logging "root" messages (outside tap/targets), Meltano does not output the run_id. It makes it impossible to get all logs of a specific run inside Datadog, which is very annoying.
Example:
{"event": "Environment 'default' is active", "level": "info", "timestamp": "2025-01-10T105731.947681Z"}
{"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,592 | INFO | root | animals stream max schema_inference_record_count hit", "level": "info", "timestamp": "2025-01-10T105732.593107Z"}
{"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,593 | INFO | tap-smoke-test.animals | Beginning full_table sync of 'animals'...", "level": "info", "timestamp": "2025-01-10T105732.593758Z"}
{"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,593 | INFO | tap-smoke-test.animals | Tap has custom mapper. Using 1 provided map(s).", "level": "info", "timestamp": "2025-01-10T105732.593930Z"}
{"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,617 | INFO | singer_sdk.metrics | METRIC: {\"type\": \"timer\", \"metric\": \"sync_duration\", \"value\": 0.024380922317504883, \"tags\": {\"stream\": \"animals\", \"pid\": 78012, \"context\": {}, \"status\": \"succeeded\"}}", "level": "info", "timestamp": "2025-01-10T105732.617847Z"}
{"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,617 | INFO | singer_sdk.metrics | METRIC: {\"type\": \"counter\", \"metric\": \"record_count\", \"value\": 10, \"tags\": {\"stream\": \"animals\", \"pid\": 78012, \"context\": {}}}", "level": "info", "timestamp": "2025-01-10T105732.618194Z"}
{"event": "Incremental state has been updated at 2025-01-10 105732.673448+00:00.", "level": "info", "timestamp": "2025-01-10T105732.673458Z"}
{"set_number": 0, "block_type": "ExtractLoadBlocks", "success": true, "err": null, "event": "Block run completed.", "level": "info", "timestamp": "2025-01-10T105732.683769Z"}
Here, all taps/targets logs have the run_id perfectly set 🎉 but the "root" logs (Environment 'default' is active, Incremental state has been updated at... Block run completed) do not have it.
We ingest all these logs in Datadog and group by run_id to check our pipelines logs, hiding these "root" logs.
### Configurations
logging.yaml :
version: 1
disable_existing_loggers: false
formatters:
structured_colored:
(): meltano.core.logging.console_log_formatter
colors: true
json: # log format for json formatted logs
(): meltano.core.logging.json_formatter
callsite_parameters: false # adds pathname, lineno, and func_name to each log entry
dict_tracebacks: true # adds the exception object that is added to each log entry
show_locals: true # enables local variable logging in tracebacks
handlers:
console: # log to the console (stderr) using structured_colored formatter, logging everything at INFO level and up
class: logging.StreamHandler
level: INFO
formatter: structured_colored
stream: "ext://sys.stderr"
meltano_log: # log everything INFO and above to a file in the project root called meltano.log in json format
class: logging.handlers.RotatingFileHandler # Changed from FileHandler
level: INFO
filename: meltano.log
formatter: json
maxBytes: 5000 # 5KB
backupCount: 1 # Keep 5 backup files
loggers:
# Very important: this removes Singer messages (aka the raw data) from the logs
meltano.core.block.extract_load:
level: INFO
# We log to console & to the meltano.log file
root:
level: DEBUG
handlers: [console, meltano_log]
meltano.yml:
## This is the base configuration file for our Meltano project ##
project_id: c35e4650-7eda-4f6e-aae8-f331ae5615a3
version: 1
# Environment
default_environment: default
environments:
- name: default
# Project settings
elt:
buffer_size: 314572800 # 300Mb
send_anonymous_usage_stats: false
python: python3.11
# Feature flags
ff:
plugin_locks_required: true
# Use uv as the default installer - much faster
venv:
backend: uv
plugins:
extractors:
- name: tap-smoke-test
variant: meltano
pip_url: git+https://github.com/meltano/tap-smoke-test.git
config:
streams:
- stream_name: animals
input_filename: https://raw.githubusercontent.com/meltano/tap-smoke-test/main/demo-data/animals-data.jsonl
loaders:
- name: target-jsonl
variant: andyh1203
pip_url: target-jsonl
config:
destination_path: jsonl_output
Command ran:
meltano run tap-smoke-test target-jsonl
Meltano version:
❯ meltano --version
meltano, version 3.6.0
meltano/meltanoGitHub
06/19/2025, 5:42 PMrun_id to Meltano logging
Issue closed as completed by edgarrmondragon
meltano/meltano