I'm dipping my toes somewhat in tap development, currently looking at Branch.io. The way it works there is that you enable daily data dumps, and then they provide an API to get your data. You're not getting the data, instead they provide links to an S3 bucket where you can fetch some gzipped CSVs they generate daily. I'm wondering if I can somehow "chain" the code I'd be writing that interacts with the Branch.io API to

tap-csv

so I don't won't to write all the code for dealing with CSVs etc.

You could chain meltano commands to do api -> s3 then s3 -> postgres

Nah, OK. And no sanctioned way of using another tap's internals as a library, like (fantasizing here...)

from tap_csv import csv_parser

Could your tap also download and persist the data to s3 directly? And then just emit the s3 url as part of its output? You could run that in combination with an arbitrary target?

meltano run tap-branchio target-postgres tap-s3 target-snowflake

or similar?

I don't think I'd be able to persist anything in a bucket I don't own. Maybe I don't understand your question...

I was thinking you download the data from the s3 url you are given, and then persist to a bucket you do own (rather than parsing / reading the csv fully).

Ah.

Not sure if copying the whole tap-csv is the way to go, but you could examine the parsing code and lift it into your tap. I would have thought it's just wrapping pythons

csv

It is

You could emit everything as text or if you have well defined streams/schemas do some type casting in your individual streams.

yeah. Maybe that is a better approach overall. Besides,

tap-csv

needs to handle different styles of CSV whereas I (hopefully) only have to deal with one. And you're right about the schemas, hadn't thought of that,.

You could stream it line by line https://kokes.github.io/blog/2018/07/26/s3-objects-streaming-python.html

I'm not sure I can access it with boto3. Guess I could try. But what I'm given by the API is a HTTP link looking like

<https://branch-exports-web.s3.us-west-1.amazonaws.com/api_export/y%3D2024/m%3D08/d%3D20/app_id%3D4DC3025>...

You could certainly add tap-csv as a dependency 🙂. It's probably gonna be pretty stable for the foreseeable future but I'd still pin it to a specific tag, since there's no contract on the Python API. In the long term, I'd like to get file encoding and decoding capabilities built into the singer-sdk, so it's easier to write, for example, parquet files that are then inserted into a dwh, or like your example extract records from an API response in CSV format.