#2753 feat: Let the stream instance inform how bookmarks and record values for the incremental replication key ought to be compared
Issue created by
edgarrmondragon
### Feature scope
Taps (catalog, state, tests, etc.)
### Description
Instead of forcing all values to be JSON-compatible strings and hope we catch all cases, we should allow the stream to define how to the values from the state and the record should be pre-processed:
sdk/singer_sdk/helpers/_state.py
Lines 221 to 231 in</meltano/sdk/commit/ae7516b8a16d625c05258570c74e242f1cf64e88|ae7516b>
| old_rk_value = to_json_compatible(progress_dict.get("replication_key_value")) |
| ----------------------------------------------------------------------------- |
| new_rk_value = to_json_compatible(latest_record[replication_key]) |
| |
| if new_rk_value is None: |
| logger.warning("New replication value is null") |
| return |
| |
| if old_rk_value is None or not check_sorted or new_rk_value >= old_rk_value: |
| progress_dict["replication_key"] = replication_key |
| progress_dict["replication_key_value"] = new_rk_value |
| return |
meltano/sdk