Channels
#announcements
Title
# announcements
a
astonishing-alarm-71586
02/06/2021, 6:28 PMDoes anyone have a suggestion on a target to write parquet files to S3?
r
ripe-musician-59933
02/08/2021, 4:27 PMThere's some related discussion here: https://gitlab.com/meltano/meltano/-/issues/2455
@adorable-horse-71609 built and has been using https://github.com/fixdauto/target-s3
a
adorable-horse-71609
02/08/2021, 4:29 PMFYI @astonishing-alarm-71586 @ripe-musician-59933 we've used it for some proof-of-concepts but not for any production pipelines. Feel free to use it but know that it may not be ready for production
a
astonishing-alarm-71586
02/08/2021, 4:31 PMThanks for the input. I'll check it out. One of the targets we were looking at for S3-CSV used pandas under the covers. Seemed like it would be easy to tweak that one if necessary to change the output from Pandas to be Parquet instead of CSV.
a
adorable-horse-71609
02/08/2021, 4:33 PMiirc we did something like that
We're still debating internally what file format to use in the lake. S3 and Athena/Spectrum are so cheap that is it worth using a file format that's hard to work with?