https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
m

michel_ebner

11/10/2023, 3:00 PM
Hi there, I am currently switching my taps from singer.io to meltano framework. And today I stumbled across the first Tap i may not be able to switch, but I would be glad to get your input or maybe this should be a request for the meltano framework. I sync data from an API which is not really well written. Therefore I need to sync all data and can not just sync changes. To be specific, I sync contracts and ledger entires. Ledger entries depend on contracts (parent child relation). Now the issue: as there is a huge amount of data, I set up multi-threading for the child context. This means, for each parent result, I start X-Threads to sync the children. Without it, the job runs days, otherwise juste hours. I found no way to do multi-threading simply using the meltano sdk. Did I miss somwthing? Could this maybe be added as a feature request? I am astonished, that no one else had this "issue" before. Glad for every input and have a nice weekend 😉