Q: Are taps automatically multi-process? Particul...
# singer-tap-developmentf
fred_reimer
02/10/2022, 3:23 PMQ: Are taps automatically multi-process? Particularly if parent-child streams, are the child streams handled in a separate process or parallelized?
fred_reimer
02/10/2022, 3:52 PMLooking at the SDK, I don't see any parallelism in the taps. Targets do have a max_parallelism, but I'm not sure how that actually works. Reason asking is that there is concern over how we can parallelize requesting data in the taps. It would seem child streams would lend themselves very naturally to parallel execution, but it does not appear that is currently the case. Is that accurate?
fred_reimer
02/10/2022, 4:00 PMWould it be as "simple" as parallelizing this: https://gitlab.com/meltano/sdk/-/blob/main/singer_sdk/streams/core.py#L1021
e
edgar_ramirez_mondragon
02/10/2022, 4:35 PMMight be simple if the API isn't rate limited simple smile
e
eric_boucher
02/10/2022, 7:37 PMSmall warning - some APIs, like GitHub, don’t like when you are using the same token to make concurrent requests
f
fred_reimer
02/14/2022, 4:12 PMSo how can we make this parallel then? Even if we use partitions, they don't appear to be run in parallel. So we would have to schedule separate jobs, and in our case spin up separate pods, for each parallel extract and load we wanted to run?