<#3565 perf(mappers): Optimize stream map expressi...
# meltano-repo-activity
g
#3565 perf(mappers): Optimize stream map expression evaluation Pull request opened by edgarrmondragon Summary After the simpleeval compatibility fix was merged in #3595, this PR contains stream map performance optimizations — a 34% improvement on
test_bench_simple_map_transforms
(547 ms → 407 ms). Optimizations 1. Pre-build static
names
dict at init time
Entries that never change across records —
config
,
__stream_name__
,
__original_stream_name__
, and
fake
— are assembled once in
__init__
into
_static_eval_names
. Previously these were re-inserted into a fresh dict inside
_eval
on every single property of every single record. 2. Build the
names
dict once per record, not once per field
The
transform_fn
closure now calls
_build_eval_names(record)
once and passes the result to every
_eval
call for that record.
_eval
only patches the
self
key (the current property's original value) before each evaluation. Previously a full
record.copy()
plus several dict assignments happened inside
_eval
for every transformed property. 3. Pre-evaluate constant expressions at init time
_is_constant_expression
inspects the parsed AST of each mapping expression. If all
Name
nodes refer to known function names (and therefore not to record variables), the expression is evaluated once during
_init_functions_and_schema
and its result is cached. For every subsequent record the cached value is used directly, bypassing simpleeval entirely. Related • Closes #3561 • danthedeckie/simpleeval#176 • #3595 (simpleeval 1.0.5 compatibility, already merged to
main
) • #3605 / #3606 (debug-log removal, merged separately — no measured performance impact per CodSpeed) Summary by Sourcery Optimize stream map expression evaluation in the mapper for improved performance while preserving existing behavior. Enhancements: • Precompute a static evaluation context for config and stream metadata and reuse it across record evaluations. • Build the expression evaluator names dictionary once per record and reuse it for all property transformations within that record. • Introduce detection and pre-evaluation of constant mapping expressions so their results can be reused without re-invoking the expression evaluator. • Extend the mapper function type aliasing to support simpleeval module wrappers alongside callables. meltano/sdk