aditya_sastry
03/09/2021, 5:51 PMaditya_sastry
03/09/2021, 5:54 PMdouwe_maan
03/09/2021, 5:56 PMadding some sort of multi-extractor to single loader to Meltano's API.Interesting idea! We just started exploring the inverse in https://gitlab.com/meltano/meltano/-/issues/2626, so we can definitely explore this in an issue as well. It would only work if the taps use the exact same streams and schemas, so it'd mostly be limited to using multiple instances of the same tap, as in your scenario.
I think in the ideal world, I'd like to have all of the extract jobs that are variants of the same tap (and only differ in account configuration?) execute as parallel Airflow tasks with the loader task downstream from them as a single operation.This would require an earlier and later Airflow task to run in parallel so that one can pipe output into the other, which is not possible as I understand it, since each task is executed separately. You wouldn't want to buffer all extractor output before it is fed into the loader, so extractor and loader would need to live in one Airflow task, as is the case today.
douwe_maan
03/09/2021, 5:57 PMdouwe_maan
03/09/2021, 5:58 PMaditya_sastry
03/09/2021, 6:00 PMaditya_sastry
03/09/2021, 6:03 PMaditya_sastry
03/09/2021, 11:05 PM