https://linen.dev logo
Join Slack
Powered by
Is there a way to increase no of requests per minu...
# singer-tap-development
s
Is there a way to increase no of requests per minutes , I have a API which will allow 1000 req/min but when I run in meltano it processes at a speed of 100 req/min , is there a way to increase this
e
Not at the moment. Is 100 req/min the rate regardless of the target it's run against?
s
it's against Snowflake, didn't try a different one but I have this query on Snowflake, might be this is killing some time too https://meltano.slack.com/archives/C069CQNHDNF/p1718308838409169
@Edgar Ramírez (Arch.dev) - do we have timeline for this to be added https://github.com/meltano/sdk/pull/2295/commits/1421227e44bcd29489ba90c8a5b279e23a3d33d0
e
Not at the moment. I did some testing with a few simple taps on my namespace but I want to be very cautious with a change like that so it's gonna require a more robust approach. I'll start thinking more about it on Monday when I'm back from PTO. Thanks for bringing it up!
ty 1
s
@Edgar Ramírez (Arch.dev) - here are few observations for this I have implemented similar but with a burst sync calls in threading for child stream with parent Id . I am facing below issue. • stream state getting race condition in deep copy used when writing it I was wondering if we run 2 stream parallel with process and they are parent and child then will child will trigger as soon as context or partition is updated for child , if so there can be a race condition here too. Is there a way to ignore writing state if we don’t have replication key defined
@Edgar Ramírez (Arch.dev) -- another observation cost is not captured self._sync_costs is always None in Parallel execution
a
Hey @Edgar Ramírez (Arch.dev) Any update on this? I have a similar case where the source allows upto 700 calls per min. When I run this with 
target-jsonl
, I am getting close to 30-40 calls per min. How to increase requests per second?
s
@ashish, I have this implemented in a tap but its a work around
can you help me with these queries • are you calling top-level API or its parent-child stream • if its parent-child stream do you need to store the state of the child
a
Hi @Siddu Hussain I am calling the top level API only.
s
And do below things • build chunks of streams , like splitting a stream into n streams and let Meltano handle it • To do this you will need a dynamic stream class and create a new stream when initiating stream calls
a
How to use this?
s
You need to replace the dependency in your poetry file in your tap
a
how many calls will it make then?
s
It completely depends on how many threads you create and how fast your source is
For my use case I was having parent child relationship which runs a bit different and simpler to implement child calls
I was able to make 800-900 calls per minute while my source can handle 1200/minute
Also consider using process over thread if you are writing code outside Meltano framework as Meltano uses few deep copies abd which will cause race condition
Considering the overload of process handling for your use case I feel 500-600 calls is a fair number with 10 threads
a
Okay. Thanks. I will check
2 Views
Open in Slack
PreviousNext
Powered by