Hi! I am currently working on a data transformation utility for Meltano and needed to test it with a target that is able to dump parquet files to S3. I found that target-s3-parquet requires an

athena_database

setting, and I don't have an Athena database so I decided to write my own target that uses pyarrow for writing plain parquet files into S3. It's quite simple so far and I haven't done much testing yet but it seems to work fine with tap-github at least. https://github.com/jkausti/target-s3

Awesome. Just looking through the code now. Spotted one problem already.

datetime.utcnow()

might not be what you think it is. It does not return a (time-zone) aware object but a naive datetime object without timezone information instead. https://docs.python.org/3/library/datetime.html#datetime.datetime.utcnow

Yeah it sure isn't what I thought it was 🫠 I will certainly fix that. Thanks for looking through the code

I think it actually works out in your use case, because

NOW.isoformat()

would otherwise include a timezone offset in the output ... depends if you want the object name itself to include a timezone offset ... https://docs.python.org/3/library/datetime.html#datetime.datetime.isoformat

The idea was to get the current datetime in the utc timezone, and then get the year, month and day based on that. Will look more into how to create the datetime object correctly and fix that 👍 I was a bit surprised myself to find that I couldn't find an existing target for writing only parquet files to S3 (without Athena/Glue). Hopefully it works for your use cases as well!

Hi can anyone update. Your target is great, but its giving error for

raise NoRegionError()