Hi all! I am noticing that if my source has large no. of records(~10M) then the initial start of the sync takes a lot of time. At one point it doesn't start at all..my logs are stuck at Initialising stream itself. Is there a way to speed this up ? running redshift sdk tap.

There's two questions in here I think

I am noticing that if my source has large no. of records(~10M) then the initial start of the sync takes a lot of time .Is there a way to speed this up ? running redshift sdk tap.

I don't know that tap specifically but there's a few ways to dive into performance tuning 1. https://sdk.meltano.com/en/latest/dev_guide.html#testing-performance generate a flame graph and look at what's taking so long in your case per record, and then looking into how to speed that up 2. Parallelizing streams (run each stream ie table in this case) seperately 3. Implement something like BATCH processing on both your tap and target see https://sdk.meltano.com/en/latest/batch.html

At one point it doesn't start at all..my logs are stuck at Initialising stream itself.

This is probably something else, and some way to replicate this is the way to go to try to find some help