Does anyone have a suggestion on a target to write...
# announcementsn
nate_atkins
02/06/2021, 6:28 PMDoes anyone have a suggestion on a target to write parquet files to S3?
d
douwe_maan
02/08/2021, 4:27 PMThere's some related discussion here: https://gitlab.com/meltano/meltano/-/issues/2455
@julian_knight built and has been using https://github.com/fixdauto/target-s3
j
julian_knight
02/08/2021, 4:29 PMFYI @nate_atkins @douwe_maan we've used it for some proof-of-concepts but not for any production pipelines. Feel free to use it but know that it may not be ready for production
n
nate_atkins
02/08/2021, 4:31 PMThanks for the input. I'll check it out. One of the targets we were looking at for S3-CSV used pandas under the covers. Seemed like it would be easy to tweak that one if necessary to change the output from Pandas to be Parquet instead of CSV.
j
julian_knight
02/08/2021, 4:33 PMiirc we did something like that
julian_knight
02/08/2021, 4:37 PMWe're still debating internally what file format to use in the lake. S3 and Athena/Spectrum are so cheap that is it worth using a file format that's hard to work with?
6 Views