<#3687 perf: Improve record and schema flattening ...
# meltano-repo-activity
g
#3687 perf: Improve record and schema flattening performance Pull request opened by edgarrmondragon • perf: Module-level regex compiled pattern • perf: Module-level
frozenset({"null","object","array"})
• perf: Avoid unnecessary key and schema copies • perf: Construct flattened field key incrementally as a string instead of a list not worth it Summary by Sourcery Improve performance of key flattening and JSON dumping by hoisting reusable constructs to module level. Enhancements: • Use a precompiled module-level lowercase regex for key reduction during flattening. • Use a shared module-level frozenset for null/object/array type comparisons in JSON dumping logic. Summary by Sourcery Optimize schema and record flattening to reduce overhead and improve performance. Enhancements: • Hoist the lowercase regex pattern and null/object/array type set to module-level constants for reuse. • Flatten schema and records using an incremental key-prefix string rather than repeatedly rebuilding key lists, while preserving duplicate-key detection semantics. • Avoid unnecessary deep copies when flattening schemas by building the output schema from the input node directly. • Replace sort-and-group duplicate key detection with an O(n) set-based check for flattened schema items. Tests: • Update snapshot JSONL outputs for mapped and flattened streams to reflect the revised flattening behavior. meltano/sdk