Hi there,
I am currently switching my taps from
singer.io to meltano framework.
And today I stumbled across the first Tap i may not be able to switch, but I would be glad to get your input or maybe this should be a request for the meltano framework.
I sync data from an API which is not really well written. Therefore I need to sync all data and can not just sync changes. To be specific, I sync contracts and ledger entires. Ledger entries depend on contracts (parent child relation).
Now the issue: as there is a huge amount of data, I set up multi-threading for the child context. This means, for each parent result, I start X-Threads to sync the children. Without it, the job runs days, otherwise juste hours.
I found no way to do multi-threading simply using the meltano sdk. Did I miss somwthing? Could this maybe be added as a feature request? I am astonished, that no one else had this "issue" before.
Glad for every input and have a nice weekend 😉