Question about ingesting data into s3 • Using docker to run the Meltano shell • Added s3 variables to Meltano env so python function below exists and runs in same ecosystem Current workflow I have set up: • Use tap to extract Salesforce objects as csv files using Meltano • Python function to read csvs and use awswrangler.to_parquet function to create parquet file in s3 bucket (dynamic folder name like S3://path/salesforce_inbound/Salesforce_objects/salesforce_lead) Question: is it possible to automate this through Meltano or through transforms? Or does this workflow even make sense?

Hey @aditya_goyal! I think you can simplify this by directly writing your salesforce output to parquet https://hub.meltano.com/targets/parquet. I dont see a parquet s3 writer though so you still would have to upload to S3 somehow.

I was having an issue where it was writing multiple objects into 1 parquet file, like streams, but will test out, thanks!

You could also fork that target and have it write those parquet files to S3 instead of local

Yeah I might try that! Thanks for the inspiration! (And checking me that I'm not going in the wrong direction!)

Hi, @aditya_goyal. I'd love to see a robust Parquet target. There are a few out in the community but I haven't tested them much. There are also considerations around replace-vs-append, where without deleting prior parquet files you'd probably be doing append-only. I would be happy to discuss this further with you. Feel free to ping me an issue link and I'll be happy to chime in there.

Thanks for sharing @aaronsteers, I think one consideration here is if going to target-s3-parquet route, that should look completely different than the target-parquet route. Going to spend some dev time over the next couple weeks thinking about which would be best to figure out for myself + company! This is giving me lots of ideas and will definitely deep dive sometime next week