#3832 feat(targets): Decode Arrow's canonical JSON extension type when reading Arrow batch files Pull request opened by edgarrmondragon Summary

ArrowReader

now decodes values of Arrow's canonical JSON extension type (

arrow.json

,

pyarrow.json_()

) into Python objects, at the top level and inside structs and lists. Sinks receive the same values a

RECORD

message would carry, instead of JSON text. This lets taps put free-form values into Arrow BATCH files without a fixed Arrow type. MeltanoLabs/tap-mongodb#190 needs it for sub-documents, arrays and fields with mixed BSON types inside a

document

struct. Without it, a target loading

document

into a JSON/VARIANT column would get those fields as strings in BATCH mode but as nested objects in RECORD mode. Details • The decoder is built from the file's schema once, so files without

arrow.json

columns take the same path as before. • Values are parsed with

singer_sdk.singerlib.json.deserialize_json

, the same parser used for

RECORD

lines, so floats become

Decimal

as they do there. • nanoarrow's

UnregisteredExtensionWarning

for

arrow.json

is silenced, since the reader now handles that extension. Warnings for other extensions are unchanged. •

schema.extension

is available in nanoarrow 0.7.0, the lowest version allowed. •

docs/batch.md

tells tap authors to use

pyarrow.json_()

for free-form values. Tests

test_arrow_reader_decodes_json

covers top-level, struct and list

arrow.json

values, nulls at each level, a list without JSON items, and the absence of the warning. It fails without the change. The new code is fully covered in the batch tests, and ruff, mypy and ty pass. https://claude.ai/code/session_01NSaQP1g64Qxf2zyvZ5Jwyh Summary by Sourcery Decode canonical JSON extension values when reading Arrow batch files so free-form data matches RECORD message semantics. New Features: • Decode Arrow's canonical

arrow.json

extension values into the same Python objects produced from RECORD messages, including values nested in structs and lists. Bug Fixes: • Ensure Arrow batch reads preserve consistent JSON value types across BATCH and RECORD modes and suppress the expected unregistered-extension warning for

arrow.json

. Documentation: • Document using

pyarrow.json_()

for free-form values in Arrow batch files. Tests: • Add coverage for top-level, nested, list, and null

arrow.json

values, including RECORD-compatible decimal parsing and warning behavior. meltano/sdk