What is best way to increase rate of target API calls? Klaviyo allows 300+ API calls / second and we are trying to optimize this tap. Currently our meltano integration is running 1/10 the speed of Stitch

To confirm, is this using a new tap-klavio built on the SDK? Also, when using stitch, were you using the "v1" connector described here? I checked the Singer github repo for their tap and I don't see any extra parallelization or async handling. We specifically logged this issue because of performance limitations mentioned as related to the Klavio API itself - but I don't have any explanation for why you'd get better performance from the Singer/Stitch version the tap, since it doesn't do any parallel or asyn work I could find. Also - assuming this is SDK-based, you might get some boost from the latest version which had some performance improvements. If you can confirm the SDK version number is the latest and some other specifics around the endpoint being synced, it would be much appreciatd.

Thanks for the help as always @aaronsteers I'm not sure if it is build off the SDK -- will check it out We're using this klaviyo tap which: https://github.com/singer-io/tap-klaviyo It is the same as the "v1" you mentioned Stitch's klaviyo tap is technically community maintained so I don't think it's a special proprietary one of theirs

Interesting... yeah, then I'd expect performance to be the same unless Stitch has a "v2" that's not public.

there's a drastic difference which makes me think we're doing something obvious wrong what would be some key info we could provide to help troubleshoot? we're currently organizing logs from comparable periods and records other integrations are very comparable with stitch

If you still have access to the Stitch environment, I think it would be helpful to have the timing data both ways, with record counts, from one isolated stream - preferably a full table sync so we don't have to worry about differing saved state and starting points.

yes we do! @tom_mcgrail and @kai_yokoyama -- let's sync on this so we can organize for the meltano team

Also helpful to know how you are landing the data in order to ensure the target isn't holding up the process.

we are using target-redshift from datamill https://github.com/datamill-co/target-redshift thanks for the tip re:

meltano invoke

, we'll compare speeds when outputting to a local file

The

datamill

variant denests everything. That usually consumes time and a fair amount of memory

got it -- even if our schema is only a limited number of columns? we noticed that stich is denesting 'receive' more than datamill but could be because of some setting here are pics

@edgar_ramirez_mondragon's explanation about denesting may also explain some of the slowness and the additional memory consumption seen on the target side.