Hi Team I have a question reg

meltano run

Does meltano ingests data asynchronously from tap to target? Or it depends on the streams/start date/end date in tap options?

Hi @ashish 👋🏼 It definitely does not strictly depend on a tap's configured start or end date. Some taps may require a

start_date

for the initial extraction, but after that the replication key picks up.

Hey @edgar_ramirez_mondragon But the ingestion to target is async right?

The word async is a bit overloaded. What do you mean by it?

Okay. Lets take a scenario. I have a tap with a few streams. The tap starts fetching data from source one stream at a time. Lets consider a particular stream has large data volume. So, • will the tap accumulate all the data for the stream first, then pass it to target? • Or, will it send a set of records (batching based on some logic) to target at the same time it is fetching more data from source?

Gotcha. Taps are usually not async, but targets normally parallelize ingestion to reduce backpressure. The preview BATCH messages help with that too, and they do essentially make ingestion async.

I saw that my stream was being ingested and at the same time I am getting records in DB (in batch of 5500). Source - tap-amazon-sp, Target - Snowflake. So can we say that taps can make data available to target before ingestion for the stream is complete?

Yes