Theoretical question, I'm loading data from ~10 di...
# announcements
a
Theoretical question, I'm loading data from ~10 different (inherited) Google Analytics taps into a single redshift schema. I have to run each ELT job to completion in a sequential manner to avoid lock conflicts on the DW, and I'm considering whether I want to refactor our google analytics tap to handle an array of account+property+view configurations to parallelize the extract operation, or whether you have interest in adding some sort of multi-extractor to single loader to Meltano's API. I think in the ideal world, I'd like to have all of the extract jobs that are variants of the same tap (and only differ in account configuration?) execute as parallel Airflow tasks with the  loader task downstream from them as a single operation. Trying to figure out how to best get there, or if we're getting outside the scope of Meltano's roadmap.
My primary concern here is that we'll soon be running 10-30 accounts across 10-15 taps, resulting in some brutal Gantt charts...
d
adding some sort of multi-extractor to single loader to Meltano's API.
Interesting idea! We just started exploring the inverse in https://gitlab.com/meltano/meltano/-/issues/2626, so we can definitely explore this in an issue as well. It would only work if the taps use the exact same streams and schemas, so it'd mostly be limited to using multiple instances of the same tap, as in your scenario.
I think in the ideal world, I'd like to have all of the extract jobs that are variants of the same tap (and only differ in account configuration?) execute as parallel Airflow tasks with the loader task downstream from them as a single operation.
This would require an earlier and later Airflow task to run in parallel so that one can pipe output into the other, which is not possible as I understand it, since each task is executed separately. You wouldn't want to buffer all extractor output before it is fed into the loader, so extractor and loader would need to live in one Airflow task, as is the case today.
I think extending tap-google-analytics to support multiple accounts is a fine solution, and it works around the "what if multiple taps use different stream schemas?" issue
Another option would be to have each extractor load into a separate Redshift schema, and then to use dbt to merge all of the tables together
a
Makes a lot of sense to me, another option that occurs to me now for the redshift specific case is buffering the taps in parallel to S3 and then running the s3copy operations at the end.
I'm going to work on extending the tap-google-analytics support to handle multiple accounts/properties/views (under a single authorizer) as a first step as well.
FYI @olya_lukashina