Hey, is there support for async streams? Or do you...
# singer-tap-development
a
Hey, is there support for async streams? Or do you consider adding this at some point? I found this only discussed In https://gitlab.com/meltano/sdk/-/issues/184 3 years ago. I am asking, because I am currently struggling with the performance of one of my child streams. My parent stream returns a lot of rows every day, leading to my child stream running for 4+ hours every day. Solutions unrelated to async streams are also appreciated.
My context right now is pulling tickets (parent) and their conversations (child) from the freshdesk api
v
What we do right now when we need to do this is add a
clean_up
routine to
sync_all
in the tap and store all the parent data into memory, then just spawn a bunch of threads and iterate through them. Not the cleanest and it only works for full_syncs but it gets the job done for us
👍 1
a
Any suggestions to https://github.com/acarter24/tap-freshdesk very welcome.
👍 1