Is there any sort of formula or back-of-the-envelope way to estimate the optimal

itersize

for extractors and

batch_size_rows

for loaders? In my case its

postgres

->

snowflake

. Or a good set of tests to run to determine the best values for each?

I don't know about the target-snowflake target. For my own target it's pretty much just a decision on how much memory do you want the target to use. Bigger batch size the faster it is (not hugely significant though past a certain size). That's because the target isn't optimized to take advantage of every thread on the machine, which I think most singer targets are basically the but I could be mistaken! The way to find out is to profile the target

That’s a good reframe of my question: how do I profile the target (in this case Snowflake?)

https://sdk.meltano.com/en/latest/dev_guide.html?highlight=vizviewer#testing-performance would need to modify this for your target but the question more generally is "how do I profile a python app"

@visch this seems more for testing a package whereas what I’m looking for are optimizing two parameters that I would expect to be dependent on the configuration of the source and target themselves. e.g. the relationship between

batch_size_rows

and Snowflake warehouse size, or upper limits on Snowflake batch reads from external and internal stages, or the relationship between

itersize

and Postgres configuration (e.g. itersize as a rule of thumb should be 1/X of total memory on the instance, stuff like that)

Right so performance on different variables, I'd guess people have better library ideas but my brain hops to https://github.com/closeio/ciso8601 Benchmark section https://github.com/closeio/ciso8601#benchmark Benchmark code https://github.com/closeio/ciso8601/blob/master/benchmarking/run_benchmarks.sh Could do something like that

I don’t think this is a code benchmarking question, this is about better understanding how Postgres and Snowflake best work