<#3779 feat: Build config schemas from capabilitie...
# meltano-repo-activity
g
#3779 feat: Build config schemas from capabilities, make `batch_config` a no-op in targets, and ignore `batch_config` without `encoding` in taps Pull request opened by edgarrmondragon Summary • Build the built-in config JSON schema per plugin type from the declared capabilities, via
tap_config_for_capabilities
,
target_config_for_capabilities
,
sql_tap_config_for_capabilities
and
sql_target_config_for_capabilities
. The existing module-level constants (
BATCH_CONFIG
,
STREAM_MAPS_CONFIG
, ...) are kept for backwards compatibility. • Split the
batch_config
schema into a tap version and a target version. Targets read the encoding off each
BATCH
message, so their
batch_config
is now a no-op: it's ignored with a warning and the schema only declares an empty object. •
Stream.get_batch_config
stays in RECORD mode when
batch_config
has no `encoding`: it logs a warning and returns
None
instead of raising
TypeError: BatchConfig.__init__() missing 1 required positional argument: 'encoding'
. • Declare the
START_DATE
capability for the dummyjson and GitLab sample taps, which already support
start_date
. Why
encoding
is not marked
required
in the schema
An earlier revision of this PR made
batch_config.encoding
and
batch_config.encoding.format
required. That is correct JSON Schema (they're only needed when
batch_config
is set), but the Meltano Hub flattens nested
required
lists into
settings_group_validation
, which turned
batch_config.encoding.format
into a mandatory setting for every user and would have enabled BATCH for everyone (meltano/hub#2334). The Hub parser is fixed separately, but nested
required
under an optional object is fragile for any settings manager, so the schema leaves
encoding
optional and the SDK handles a missing
encoding
at run time. This also protects users whose lock files hold partial
batch_config
values (for example
batch_size
and
storage.root
published as setting defaults) from a crash before the first record, see MeltanoLabs/tap-snowflake#117. Notes • Taps pinned to SDK 0.54 still raise on a
batch_config
without
encoding
, so they need their own guard (see MeltanoLabs/tap-snowflake#130). • The
API Changes
check reports the module-level config constants as "value changed" because they are now assigned from
_*_config()
builders instead of literal
PropertiesList(...)
expressions. The names and resulting schemas are unchanged apart from the
batch_config
changes above. Testing • Added tests for the capability-specific tap, target, SQL tap and SQL target schemas. • Added tests for
get_batch_config
with a missing, empty and partial
batch_config
(RECORD mode, with a warning) and with an
encoding
. • Added a check that no
required
list appears under the tap
batch_config
schema. • Updated the about-info snapshots (core and package taps). Related • MeltanoLabs/tap-snowflake#121 • MeltanoLabs/tap-snowflake#130 • meltano/hub#2334 meltano/sdk