#2121 Parallelize taps by partition (configurable in meltano.yml) Issue created by melgazar9 Feature scope Taps (catalog, state, stream maps, tests, etc.) Description I'm considering writing my own parallelized task for a meltano tap. I'm submitting this feature request since this can be generalized to run taps in parallel by partition (or even by state, but it's less obvious how to do that IMO). I think we can use the

multiprocessing

library to run multi-processes by partition. Default behavior of meltano taps would run unless both of these conditions are set: • There is a partition set in the tap • The setting

parallelize

or

multiprocessing

(or

multithreading

??) is set in

meltano.yml

Something like this:

from multiprocessing import Process

processes = []
for i in range(<meltano processes set in meltano.yml>):
    p = Process(target=<meltano run command for specific partition>, args=args)
    p.start()
    processes.append(p)

for process in processes:
    process.join()

or similarly:

with concurrent.futures.ProcessPoolExecutor() as executor:
    results = [<meltano run command> for cmd in range(<n processes in meltano.yml>)]
    
    for r in concurrent.futures.as_completed(results):
        r.result()

I think you can also do something like this:

executor.map(processes, args)

meltano/sdk