Hello Community
Wanted to know can we speed up the...
# troubleshootingm
monika_rajput
12/09/2021, 1:54 PMHello Community
Wanted to know can we speed up the data extraction and loading of meltano pipeline?
a
aaron_phethean
12/09/2021, 2:12 PMHi Monika, confident someone will be able to help! But I'm pretty sure the first questions will be
1. which tap
2. which target
3. how are you executing the pipelines?
m
monika_rajput
12/15/2021, 11:46 AM1. tap-klaviyo
2. pipelinewise redshift
3. Running pipeline via UI
monika_rajput
12/15/2021, 11:47 AMSorry for the late response.
monika_rajput
12/15/2021, 11:47 AM@aaron_phethean
monika_rajput
12/15/2021, 5:02 PM@edward_ryan
e
edward_ryan
12/15/2021, 5:05 PMTo be specific, our tap is moving at reasonable rate
However our loads seem to take a little longer
Wondering if there's a best practice for Redshift data loads
a
aaron_phethean
01/10/2022, 8:42 PMHi @edward_ryan @monika_rajput completely missed this thread - I need to improve my Slack skills! We are using Postgres currently so no experience of Redshift. I'd be tempted to narrow down the problem. Eg can you load a reasonable input straight into the target without anything else except the target and Redshift.
1. Run the tap > file
2. Run the cat file > target
3. Tune and repeat step 2
In the past really challenging performance scenarios boiled down to disk or network. Network buffer sizes can have a huge impact.