In continuation of <https discuss meltano com t 16193315 gen Meltano #troubleshooting

In continuation of <the discussion> on general per...

Denis I.

01/04/2024, 10:31 AM

In continuation of the discussion on general performance of Meltano run command: @Edgar Ramírez (Arch.dev) / @edgar_ramirez_mondragon (sorry, I’m not sure which account to tag) I’ve ran a simple test changing dotenv_env from

@property

@cached_property

and it shows a significant improvement on overall execution time for my multiple jobs: for a tap with a low self execution time (it just checks a replication slot for new data on an idle db) I see 5x execution time improvement.

🤯 1

👀 2

Denis I.

01/04/2024, 10:31 AM

@property

21.16s user 0.47s system 96% cpu 22.457 total 22.10s user 0.51s system 95% cpu 23.714 total 21.99s user 0.50s system 96% cpu 23.402 total avg 23.191 total

@cached_property

3.12s user 0.31s system 77% cpu 4.436 total 3.00s user 0.29s system 76% cpu 4.281 total 3.10s user 0.30s system 75% cpu 4.523 total avg 4.413 total

Denis I.

01/04/2024, 3:53 PM

And as expected it breaks some tests 😊

Copy code

FAILED tests/meltano/core/plugin/test_plugin_settings.py::TestPluginSettingsService::test_store_dotenv - AssertionError: assert (None, <Setti...T: 'default'>) == (False, <Sett...NV: 'dotenv'>)
FAILED tests/meltano/core/tracking/test_tracker.py::TestTracker::test_timeout_if_endpoint_unavailable[no_timeout] - assert True is False
FAILED tests/meltano/core/plugin/test_plugin_settings.py::TestPluginSettingsService::test_namespace_as_env_prefix - AssertionError: assert 'name_prefix' == 'namespace_prefix'
FAILED tests/meltano/core/plugin/test_plugin_settings.py::TestPluginSettingsService::test_date_values - AssertionError: assert 'THIS_IS_FROM_DOTENV' == '2024-01-04'
FAILED tests/meltano/core/plugin/test_plugin_settings.py::TestPluginSettingsService::test_as_env - AssertionError: assert 'THIS_IS_FROM_DOTENV' == 'mock'
FAILED tests/meltano/core/plugin/test_plugin_settings.py::TestPluginSettingsService::test_env_var_expansion - AssertionError: assert '  ' == 'rock paper scissors'

Edgar Ramírez (Arch.dev)

01/04/2024, 4:34 PM

Thanks for confirming @Denis I.! I created https://github.com/meltano/meltano/issues/8342 and have a prelim branch in https://github.com/meltano/meltano/tree/edgarrmondragon/perf/cache-project-dotenv-contents.

🥳 1

Denis I.

01/04/2024, 7:08 PM

Thank you @Edgar Ramírez (Arch.dev)! Launched the py-spy again to compare the differences

speedscope_non_cached.json speedscope_cached.json

🙌 1

Edgar Ramírez (Arch.dev)

01/05/2024, 4:28 AM

Thanks Denis! I do see the improvement there 😁

😁 1

Denis I.

01/05/2024, 11:13 AM

I’m really impressed about the performance improvement! Ping me if I can be of any assistance here

Open in Slack

Previous Next