#3400 WIP: Redesign sink architecture
Pull request opened by
edgarrmondragon on <!date^1764738898^{date_short}|2025-12-03T05:14:58Z>
Summary by Sourcery
Introduce a modern, type-safe sink loading architecture while preserving backwards compatibility with existing dict-based sink implementations.
New Features:
• Add load_batch and load_record entry points as the recommended modern APIs for batch and record sinks, alongside legacy process_* methods.
• Introduce BatchContext as a strongly-typed batch context object, including helpers to convert to and from legacy dict contexts.
• Provide pluggable batch accumulation strategies (count, time window, size, hybrid, and no-batch) via a new BatchStrategy abstraction.
• Expose new sink protocols (IRecordLoader, IBatchLoader, IPreprocessable, IVersionable, IBatchable) for clearer, typed interfaces and extension points.
• Extend the CSV target sink to demonstrate the modern load_batch pattern using BatchContext.
Enhancements:
• Delay SQL sink table preparation to the first batch rather than during setup to avoid data loss when multiple schema messages arrive for the same stream.
• Export the new BatchContext, batch strategy classes, and sink protocol interfaces from the sinks package for easier consumption.
• Improve sink base class docs and behavior so that subclasses must override either the modern load_* methods or the legacy process_* methods, with clearer error messages and guidance.
Tests:
• Add tests covering modern vs legacy load methods, BatchContext conversion, and error behavior when sinks fail to override any loading method.
meltano/sdk