#3787 bug: `_write_state_message` cannot serialize `decimal.Decimal` Issue created by dmzobel Singer SDK Version 0.54.5 Is this a regression? • Yes Python Version 3.11 Bug scope Taps (catalog, state, etc.) Operating System Linux, MacOS Description

TargetBase._write_state_message

serializes state with plain

json.dumps

, which has no handler for

decimal.Decimal

. The SDK's Singer-message reader (`simplejson`/`parse_float=decimal.Decimal`) converts every float in every incoming message (RECORD and STATE) to

Decimal

to preserve precision for money-like fields. Because of this, any tap that ever puts a plain float into a STATE message will crash the target the next time it re-serializes that state (e.g. via the periodic max-record-age drain). This is the same underlying defect as #1825 (

Object of type Decimal is not JSON serializable

, closed via #1826), but in a different call site. #1826 only patched `singer_sdk/batch.py`'s `JSONLinesBatcher.get_batches`: - gz.writelines((json.dumps(record) + "\n").encode() for record in chunk) + gz.writelines((json.dumps(record, default=str) + "\n").encode() for record in chunk)

singer_sdk/target_base.py::_write_state_message

was never touched and, as of 0.54.6 (current latest), does not include

default=

handler: def _write_state_message(self, state: dict) -> None: ... state_json = json.dumps(state) Reproduction I found this using the singer-io variant of tap-shopify, and Snowflake as the loader. However, repro doesn't require Snowflake specifically; any

singer_sdk

-based target reproduces this, since the bug is in the shared

TargetBase

class: 1. A tap emits a STATE message where some bookmark/metadata value is a bare JSON float that is not a `key_properties`/record field (e.g.

{"bookmarks": {"my_stream": {"some_cursor_metadata": 30.0}}}

). 2. The target's message reader parses that

30.0

into

Decimal("30.0")

and stores it as part of

self._latest_state

. 3. Anything that calls

drain_all()

before end-of-stream — most commonly

_handle_max_record_age()

— calls

_write_state_message(copy.deepcopy(self._latest_state))

, which crashes. Real-world trigger we hit `tap-shopify`'s

Orders

stream uses Shopify's async Bulk Operation API. Once a bulk operation completes, it calls (

tap_shopify/streams/orders.py

): self.update_bookmark( bookmark_value=utils.strftime(current_bookmark), bulk_op_metadata={ "bulk_operation_id": op.get("id"), "status": current_status, "created_at": op.get("createdAt"), "last_date_window": self.date_window_size, # <- a bare float, e.g. 30.0 } )

self.date_window_size

is

float(config.get("date_window_size") or DEFAULT)

. This triggers the serialization bug. Traceback

Traceback (most recent call last):
  File ".../target-snowflake/venv/bin/target-snowflake", line 10, in <module>
    sys.exit(TargetSnowflake.cli())
  File ".../click/core.py", line 1631, in __call__
    return self.main(*args, **kwargs)
  File ".../click/core.py", line 1552, in main
    rv = self.invoke(ctx)
  File ".../click/core.py", line 1415, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ".../click/core.py", line 910, in invoke
    return callback(*args, **kwargs)
  File ".../singer_sdk/plugin_base.py", line 142, in invoke
    return super().invoke(ctx)
  File ".../singer_sdk/target_base.py", line 583, in invoke
    target.listen(file_input)
  File ".../singer_sdk/plugin_base.py", line 767, in listen
    counter = self.process_lines(file_input)
  File ".../singer_sdk/plugin_base.py", line 793, in process_lines
    return self.message_reader.process_lines(...)
  File ".../singer_sdk/singerlib/encoding/base.py", line 72, in process_lines
    callback(line_dict)
  File ".../singer_sdk/target_base.py", line 350, in _process_record_message
    self._handle_max_record_age()
  File ".../singer_sdk/target_base.py", line 296, in _handle_max_record_age
    self.drain_all()
  File ".../singer_sdk/target_base.py", line 490, in drain_all
    self._write_state_message(copy.deepcopy(self._latest_state))
  File ".../singer_sdk/target_base.py", line 529, in _write_state_message
    state_json = json.dumps(state)
  File ".../json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File ".../json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File ".../json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
  File ".../json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Decimal is not JSON serializable

Expected behavior State messages containing

Decimal

values (introduced by the SDK's own

parse_float=Decimal

reader) should serialize without error, the same way

singer_sdk/batch.py

was fixed to in #1826. Suggested fix Apply the same treatment used in #1826 to

_write_state_message

(and audit for any other unguarded

json.dumps

calls elsewhere in the SDK that consume reader-parsed messages, since this is the second call site to hit the identical issue independently): def _write_state_message(self, state: dict) -> None: ... - state_json = json.dumps(state) + state_json = json.dumps(state, default=str) Environment • `singer-sdk`: 0.54.5 (reproduced), confirmed still present in latest 0.54.6 • `meltanolabs-target-snowflake`: 0.19.2 (reproduced), confirmed still present in latest 0.20.0 — this target does not override

_write_state_message

, so it inherits the bug unmodified from

TargetBase

•

tap-shopify

(singer-io variant): 3.12.1 — included only as the tap that happens to trigger this particular instance; the underlying defect is in the SDK's target base class and is reachable from any tap that ever stores a non-money float in state Link to Slack/Linen No response meltano/sdk