Hello, I'm looking for a target for S3/Athena that...
# plugins-general
a
Hello, I'm looking for a target for S3/Athena that supports the output format as
parquet
and compression as
zstd
. I've found these in Meltano Hub, but all seem to lack these specific configs: • target-athena: no support for
parquet
format • target-parquet: no support for
S3
target-s3-parquet: no support for
zstd
compression Anyone else with the same use case that could help here? Thanks in advance!
v
A "universal" target is really something that people would enjoy for files Formats:
jsonl
avro
csv
parquet
xlsx
singer
etc Compression:
zip
,
bz2
,
gzip
,
lzma
,
xz
, then adding
zstd
wouldn't be major I would think Protocols:
s3
gcs
local
sftp
ftp
ftps
, etc We started on this with this tap https://github.com/MeltanoLabs/tap-universal-file , but a target with the same idea would be great Haven't seen one that goes for this but I haven't dove too deep looking!
a
During my searches today also found that
tap
! It looks very promising!