Discussion about working on target-s3 (all) 🧵

@pat_nadolny — what repo should this live in? personal repo to start? I’ve got a good chunk of foundational code for the

parquet

format (and pattern for future formats) here — still lots of tweaking needed, but I think this will be functional for me at the moment, feedback welcome 🙂

Hi, @andy_crowe - thanks for your comment. I don't know if either system can natively generate the

parquet

file format.

DuckDB

or

arrow

could in theory be used to generate the Parquet dataset, then

smart_open

or

PyFilesystem

could be used to upload/write the file bytes to the respective cloud. I've added a comment to this effect to my new issue proposal for a generic multi-cloud target. To be clear, there may be other challenges I've not foreseen. In total, the dev effort could be significant... but I do feel that it would be a valuable addition to our currently available targets if there's a path forward here.

what repo should this live in? personal repo to start?

@andy_crowe awesome to hear you've made progress! Its up to you really. Many people leave them in their own personal or organization's github repo. But if you dont want it in your personal repo, we've created MeltanoLabs for that exact reason (see this blog post for what we view as the connector ownership models) , you have the option to have the repo live there but you'd still be the primary maintainer. Or you could always wait and migrate it out of your personal namespace down the line. Its up to you

Awesome, thank you @pat_nadolny! Yes, this is the primary target we use in production (parquet/json). However, I’ve been using/modifying the target in our private repository — I’ll can get the latest pushed to GitHub, I think that will solve a couple of the issues you submitted

@andy_crowe thanks for the update! I'm able to get json working now but I'm still having trouble with parquet but its slightly different now, I'll open an issue to describe it.