<#3829 feat(taps): Streams can yield native Arrow ...
# meltano-repo-activity
g
#3829 feat(taps): Streams can yield native Arrow data for Arrow BATCH files via `Stream.get_record_batches` Pull request opened by edgarrmondragon Closes #3830 Summary by Sourcery Enable streams to write source-produced Arrow data directly to Arrow BATCH files while retaining existing record-based batching paths. New Features: • Allow streams to yield native Apache Arrow tables or record batches directly into Arrow BATCH files through
get_record_batches
. • Support Arrow batch rechunking, schema-aware column selection, incremental bookmark advancement, STATE emission, and partition-aware processing for native Arrow data. Enhancements: • Preserve existing record-based batching and child-stream behavior when native Arrow data is unavailable or unsupported. Documentation: • Document how to implement native Arrow data production, schema requirements, batching behavior, and bookmark handling. Tests: • Add coverage for Arrow rechunking, schema promotion, incremental and partitioned syncs, column selection, fallback behavior, child streams, and missing dependencies. meltano/sdk