#9506 feat: Parse structured logs coming from the Singer SDK
Pull request opened by
edgarrmondragon
## Description
## Related Issues
•
meltano/hub#2114
•
meltano/sdk#3279
•
#9506
## Summary by Sourcery
Add structured logging support for Singer SDK in Meltano by introducing pluggable log parsers and integrating them into plugin output handling.
New Features:
• Introduce LogParser classes (SingerSDK and Passthrough) and a LogParserFactory for structured JSON log parsing.
• Extend OutputLogger and Out to accept a log_parser parameter and attempt structured parsing before fallback logging.
• Add get_log_parser method to ProjectPlugin and a _log_parser setting to enable per-plugin log parser configuration.
Enhancements:
• Refactor SingerPlugin logging setup to switch between basic and structured SDK logging configurations.
• Propagate log_parser through plugin invocations and extract/load blocks for consistent handling.
• Define a new constant LOG_PARSER_SINGER_SDK and use SettingDefinition for _log_parser in Singer plugins.
Tests:
• Add comprehensive integration and unit tests covering parser selection, parsing behavior, and fallback logic in OutputLogger and ProjectPlugin.
## Summary by Sourcery
Add structured logging support for Singer SDK by introducing a pluggable log parser framework and integrating it into plugin output handling
New Features:
• Provide a log parsing framework with base LogParser, ParsedLogRecord dataclass, LogParserFactory, SingerSDK and Passthrough parsers
• Allow OutputLogger/Out to accept a log_parser parameter and parse written lines before falling back to standard logging
• Add a per-plugin _log_parser setting and PluginInvoker.get_log_parser method to configure structured parsing
Enhancements:
• Refactor SingerPlugin to emit structured or plain JSON logging configurations based on the selected log parser
• Propagate log_parser through plugin_command_invoker, extract_load blocks, and I/O linking
• Define LOG_PARSER_SINGER_SDK constant and register it in plugin settings definitions
Tests:
• Add unit tests for parser implementations, factory behavior, OutputLogger parsing and fallback logic, and plugin log parser configuration
• Add integration tests validating end-to-end structured logging support in PluginInvoker and ProjectPlugin
meltano/meltano