kyle_dayne
08/14/2022, 4:47 PMalexander_butler
08/14/2022, 5:26 PMaaronsteers
08/15/2022, 2:04 AMis 1mb per file good? 10mb? 100mb?It might help to know more details of your use case, but in general I'd say definitely aim to have files >10MB. For reference, Snowflake uses internal "micropartition files" around 16MB compressed, which is roughly 50-500MB uncompressed. https://stackoverflow.com/a/67945808 You can certainly use larger files but anything smaller and the overhead of processing files starts to compete with the cost of reading the data.