FYI @astonishing-alarm-71586@ripe-musician-59933 we've used it for some proof-of-concepts but not for any production pipelines. Feel free to use it but know that it may not be ready for production
02/08/2021, 4:31 PM
Thanks for the input. I'll check it out. One of the targets we were looking at for S3-CSV used pandas under the covers. Seemed like it would be easy to tweak that one if necessary to change the output from Pandas to be Parquet instead of CSV.
02/08/2021, 4:33 PM
iirc we did something like that
We're still debating internally what file format to use in the lake. S3 and Athena/Spectrum are so cheap that is it worth using a file format that's hard to work with?