#3824 feat: Targets can now read Arrow batches automatically and decode individual records from them
Pull request opened by
edgarrmondragon
SSIA
Relates to
•
#3696
Summary by Sourcery
Enable targets to automatically read supported batch files and decode their records through extensible format-specific readers.
New Features:
• Add automatic decoding of JSONL, Parquet, and Arrow batch files through registered batch readers.
• Support Arrow IPC batch files as a recognized batch format.
Bug Fixes:
• Provide clear, actionable errors when optional batch-format dependencies are missing.
Enhancements:
• Refactor batch file processing to use a common reader interface for decoding records.
• Centralize optional dependency loading for batch readers and writers.
Documentation:
• Document the new missing-dependency error in the SDK error hierarchy.
Tests:
• Add coverage for batch reader selection, format decoding, round trips, invalid Arrow files, and missing dependencies.
meltano/sdk