Meltano

I have a bucket in AWS S3 full of `.zip` and `.gz` files. Each of them will contain `csv` files. I'm wondering if there's a way for Meltano to handle these compressed files prior to using an extractor like `tap-s3-csv`

you could use a utility or a bash script.

We've also thought about it some here <https://github.com/MeltanoLabs/tap-universal-file>

Give it a shot!

:point_up: that and also <https://hub.meltano.com/extractors/tap-spreadsheets-anywhere/|https://hub.meltano.com/extractors/tap-spreadsheets-anywhere/> should handle both S3 and gzip compression

So at present it doesn't handle a compression that involves multiple files, correct?

Ah I see. No I don't think zip "balls" are well supported by any tap. I know the tap I linked uses `smart_open` so it might be worth checking if that library supports extracting multiple files from a zip file with something like a glob expression as input.

This is excellent! Thanks so much for the help. Thankfully 99% of the files are `.gz`. These extractors are awesome

<@U06CCB0EUBC> I'm going to try out both extractors. I think the Spreadsheets-Anywhere one is nice because it's closer in alignment with the `tap-s3-csv` one I already use, but I'm curious to see how `tap-universal-file`  would handle those zips. Any good spot I can start a thread if I have questions related to your singer tap? <#C06A1MD6A6L|>?

<#C06A1MD6A6L|> works :smile:

great to see you try it! I hope it is easier to change and work with but we'll see how it goes for you!

Hey <@U06CASRQ0H0> Right now I'm finding my `tap-spreadsheets-everywhere` just kinda hangs when I try to extract a csv from S3.
```meltano invoke tap-spreadsheets-anywhere --dev
2024-06-14T18:14:55.056767Z [info     ] Environment 'dev' is active
INFO Generating catalog through sampling.
INFO Found credentials in environment variables.```
Any idea what might be happening? I assume this is a fairly common problem related to listing all objects in S3? I've tried to use a very specific search pattern.

Have you tried narrowing down the search pattern to match one or at most a few files? That way we can rule issues with listing objects.

Got it, just didn't see the `search_prefix` listed notably in the `README.md` :slightly_smiling_face: All good