Does anyone have any suggestions for tracking historic pulls using Meltano or if it is better done in Airflow and if I should look into their documentation?
Specifically being able to track progress and status of ongoing historic jobs (in my case scheduled in Meltano using the Airflow Orchestrator)
@edward_ryan
d
douwe_maan
05/03/2021, 11:19 PM
That'd be a task for Airflow. Meltano tells Airflow what scheduled pipelines to run, and handles the job of extracting and loading inside the pipeline, but Airflow manages the pipeline itself
douwe_maan
05/03/2021, 11:19 PM
And this question is more appropriate for #C01UTUSP34M 🙂