#3761 feat(taps): add EndOfStreamError to skip partitions and streams gracefully Pull request opened by daria-hrabar Closes #280 Summary Introduces

EndOfStreamError

, a new exception in

singer_sdk/exceptions.py

that tap developers can raise inside

get_records()

to signal that a partition or stream should be skipped without aborting the entire sync. The SDK catches this signal — along with all other

SkippableSyncError

subclasses — at two levels: inside the

get_records

iteration in

Stream._sync_records

and around each stream call in

Tap.sync_all

. On a skippable error, a warning is logged, partition state is finalized, and the sync continues to the next partition or stream. Motivation Currently, any unhandled exception raised during a partitioned sync aborts the entire tap run. All remaining partitions and streams are silently skipped, and no state is emitted for them. This is particularly painful for taps like

tap-github

that iterate over a list of repositories as partitions — a single persistent API error on one repository prevents all subsequent repositories from being synced (see MeltanoLabs/tap-github#52). Changes •

singer_sdk/exceptions.py

— added

EndOfStreamError(SkippableSyncError)

and registered it in

__all__

•

singer_sdk/streams/core.py

— wrapped

get_records()

and its iteration loop in

Stream._sync_records

with

try/except SkippableSyncError

(skip, finalize state, continue) and

except RetriableSyncError

(finalize state, re-raise); added

RetriableSyncError

and

SkippableSyncError

to imports; updated docstring •

singer_sdk/tap_base.py

— wrapped

stream.sync()

in

Tap.sync_all

with

try/except SkippableSyncError

, logging a warning and calling `continue`; added

SkippableSyncError

to imports •

tests/core/test_streams.py

— added 11 unit tests covering partition-level skip, stream-level skip, exception hierarchy, edge cases (empty partition list, empty stream list, all/first/last/only partition failing), and verification that non-

SkippableSyncError

exceptions still propagate uncaught at both levels Design decisions •

EndOfStreamError

is an explicit, opt-in signal for tap developers. Raising it inside

get_records()

signals that a partition is intentionally skippable. Plain exceptions (e.g.

RuntimeError

) still crash the tap — this is intentional, as they may indicate genuine bugs rather than expected, recoverable conditions. • The catch uses

SkippableSyncError

rather than

EndOfStreamError

directly, so existing subclasses such as

SkippableAPIError

and

InvalidRecord

also benefit from the skip-and-continue behavior without requiring tap code changes. •

RetriableSyncError

is caught at the partition level to finalize and flush state before re-raising, ensuring the orchestrator has a clean checkpoint before the retry mechanism takes over. It is not swallowed. •

FatalSyncError

and its subclasses (including

FatalAPIError

) are intentionally left uncaught. These signal conditions severe enough to abort the entire sync — such as missing key properties or invalid stream sort order — and silently skipping them would hide genuine data integrity issues. • The

try

block wraps both the

get_records()

call and the iteration loop over it. This is necessary because

get_records

is a generator function — calling it does not execute any of its code. The

EndOfStreamError

is only raised during iteration, not at the point of the call. • The stream-level catch in

Tap.sync_all

is separate from the partition-level catch in

_sync_records

because a stream can raise

SkippableSyncError

before partition iteration begins — both levels are needed to cover different failure points in the sync lifecycle. • A structured error-reporting mechanism for skipped partitions (beyond warning logs) is not included in this PR. Input from maintainers on the preferred shape of such a mechanism is welcome before implementation. Testing All 855 tests pass across Python 3.10–3.14 with no regressions.

singer_sdk/exceptions.py

reaches 100% coverage. Summary by Sourcery Allow taps to skip explicitly skippable partitions and streams while continuing synchronization and preserving state. New Features: • Add EndOfStreamError as an explicit signal for safely skipping unrecoverable partitions or streams during a sync. Bug Fixes: • Prevent skippable partition or stream failures from aborting the remaining synchronization work while preserving state checkpoints. • Finalize and flush partition state before propagating retriable synchronization errors. Enhancements: • Handle all SkippableSyncError subclasses consistently at both partition and stream synchronization levels while allowing unrelated and fatal errors to propagate. Tests: • Add comprehensive coverage for partition and stream skipping, state-safe continuation, edge cases, exception inheritance, and propagation of non-skippable errors. meltano/sdk