#3687 perf: Improve record and schema flattening performance
Pull request opened by
edgarrmondragon
•
perf: Module-level regex compiled pattern
•
perf: Module-level frozenset({"null","object","array"})
•
perf: Avoid unnecessary key and schema copies
•
perf: Construct flattened field key incrementally as a string instead of a list not worth it
Summary by Sourcery
Improve performance of key flattening and JSON dumping by hoisting reusable constructs to module level.
Enhancements:
• Use a precompiled module-level lowercase regex for key reduction during flattening.
• Use a shared module-level frozenset for null/object/array type comparisons in JSON dumping logic.
Summary by Sourcery
Optimize schema and record flattening to reduce overhead and improve performance.
Enhancements:
• Hoist the lowercase regex pattern and null/object/array type set to module-level constants for reuse.
• Flatten schema and records using an incremental key-prefix string rather than repeatedly rebuilding key lists, while preserving duplicate-key detection semantics.
• Avoid unnecessary deep copies when flattening schemas by building the output schema from the input node directly.
• Replace sort-and-group duplicate key detection with an O(n) set-based check for flattened schema items.
Tests:
• Update snapshot JSONL outputs for mapped and flattened streams to reflect the revised flattening behavior.
meltano/sdk