Hi everyone, I was wondering if anyone is working ...
# singer-tap-developmentj
jules_huisman
11/15/2021, 3:31 PMHi everyone, I was wondering if anyone is working on (or has a) target-s3-parquet? I did find target-s3 which can save files to Parquet, but this one is not build using the SDK, and is lacking some features.
f
fred_reimer
11/15/2021, 7:30 PMTook a look. There's about 50 lines, many of which are log messages, that converts the json files locally created to parquet format. That could be added to target-athena as another file format. It already supports csv and json, and the conversion and a small change to the DDL to create the athena / glue tables is all that would be required. There's #C01RKUVUG4S
j
jules_huisman
11/15/2021, 8:33 PMYes, you are right, it was quite straight forward, I created my own version for the moment.
(Realized after I posted that it was the tap channel 🙃)
a
andrew_stewart
11/16/2021, 11:06 PMa parquet format would be a very welcome contribution to
target-athenac
charley_guillaume
01/06/2022, 12:58 PMWe are developing one in-house. It will be open-sourced when fully tested. WEe still have an issue with null columns that do not have the proper data type from the tap schema.