<#9005 feature: Add the `run_id` to Meltano loggin...
# meltano-repo-activity
g
#9005 feature: Add the
run_id
to Meltano logging
Issue created by florian-ernst-alan ### Feature scope CLI (options, error messages, logging, etc.) ### Description ### The problem When logging "root" messages (outside tap/targets), Meltano does not output the
run_id
. It makes it impossible to get all logs of a specific run inside Datadog, which is very annoying. Example: {"event": "Environment 'default' is active", "level": "info", "timestamp": "2025-01-10T105731.947681Z"} {"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,592 | INFO | root | animals stream max schema_inference_record_count hit", "level": "info", "timestamp": "2025-01-10T105732.593107Z"} {"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,593 | INFO | tap-smoke-test.animals | Beginning full_table sync of 'animals'...", "level": "info", "timestamp": "2025-01-10T105732.593758Z"} {"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,593 | INFO | tap-smoke-test.animals | Tap has custom mapper. Using 1 provided map(s).", "level": "info", "timestamp": "2025-01-10T105732.593930Z"} {"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,617 | INFO | singer_sdk.metrics | METRIC: {\"type\": \"timer\", \"metric\": \"sync_duration\", \"value\": 0.024380922317504883, \"tags\": {\"stream\": \"animals\", \"pid\": 78012, \"context\": {}, \"status\": \"succeeded\"}}", "level": "info", "timestamp": "2025-01-10T105732.617847Z"} {"consumer": false, "producer": true, "string_id": "tap-smoke-test", "cmd_type": "elb", "run_id": "cd18aa90-16f2-432a-9394-c10a98cd2a14", "job_name": "default:tap-smoke-test-to-target-jsonl", "stdio": "stderr", "name": "tap-smoke-test", "event": "2025-01-10 115732,617 | INFO | singer_sdk.metrics | METRIC: {\"type\": \"counter\", \"metric\": \"record_count\", \"value\": 10, \"tags\": {\"stream\": \"animals\", \"pid\": 78012, \"context\": {}}}", "level": "info", "timestamp": "2025-01-10T105732.618194Z"} {"event": "Incremental state has been updated at 2025-01-10 105732.673448+00:00.", "level": "info", "timestamp": "2025-01-10T105732.673458Z"} {"set_number": 0, "block_type": "ExtractLoadBlocks", "success": true, "err": null, "event": "Block run completed.", "level": "info", "timestamp": "2025-01-10T105732.683769Z"} Here, all taps/targets logs have the
run_id
perfectly set 🎉 but the "root" logs (
Environment 'default' is active
,
Incremental state has been updated at...
Block run completed
) do not have it. We ingest all these logs in Datadog and group by
run_id
to check our pipelines logs, hiding these "root" logs. ### Configurations logging.yaml : version: 1 disable_existing_loggers: false formatters: structured_colored: (): meltano.core.logging.console_log_formatter colors: true json: # log format for json formatted logs (): meltano.core.logging.json_formatter callsite_parameters: false # adds
pathname
,
lineno
, and
func_name
to each log entry dict_tracebacks: true # adds the
exception
object that is added to each log entry show_locals: true # enables local variable logging in tracebacks handlers: console: # log to the console (stderr) using structured_colored formatter, logging everything at INFO level and up class: logging.StreamHandler level: INFO formatter: structured_colored stream: "ext://sys.stderr" meltano_log: # log everything INFO and above to a file in the project root called meltano.log in json format class: logging.handlers.RotatingFileHandler # Changed from FileHandler level: INFO filename: meltano.log formatter: json maxBytes: 5000 # 5KB backupCount: 1 # Keep 5 backup files loggers: # Very important: this removes Singer messages (aka the raw data) from the logs meltano.core.block.extract_load: level: INFO # We log to console & to the meltano.log file root: level: DEBUG handlers: [console, meltano_log] meltano.yml: ## This is the base configuration file for our Meltano project ## project_id: c35e4650-7eda-4f6e-aae8-f331ae5615a3 version: 1 # Environment default_environment: default environments: - name: default # Project settings elt: buffer_size: 314572800 # 300Mb send_anonymous_usage_stats: false python: python3.11 # Feature flags ff: plugin_locks_required: true # Use uv as the default installer - much faster venv: backend: uv plugins: extractors: - name: tap-smoke-test variant: meltano pip_url: git+https://github.com/meltano/tap-smoke-test.git config: streams: - stream_name: animals input_filename: https://raw.githubusercontent.com/meltano/tap-smoke-test/main/demo-data/animals-data.jsonl loaders: - name: target-jsonl variant: andyh1203 pip_url: target-jsonl config: destination_path: jsonl_output Command ran:
Copy code
meltano run tap-smoke-test target-jsonl
Meltano version:
Copy code
❯ meltano --version
meltano, version 3.6.0
meltano/meltano