Hello all - I am exploring Meltano as a way to upgrade my company's existing EL pipeline. The current process consists of cron jobs running on k8s calling custom Python code that extracts data from a Postgres application db and loads to a Postgres data warehouse. My vision is to replace this with Airflow scheduling Meltano jobs! I have successfully configured Meltano to run for two selected tables. During my experimentation, I noticed that when a failure occurs on one table extract, then the extracts after that don't appear to run. In production, is it best practice to separate out each source table as it's own Meltano job in order to avoid one extract failure from affecting the other extracts? A couple of setups that I could imagine: • one Airflow dag on an hourly schedule that runs several Meltano jobs (one per source table) in parallel. one Airflow dag on a daily schedule that runs other Meltano jobs (one per source table) • one Airflow dag for each Meltano job, although maybe that is over-modularizing it. If anyone has experience or strong convictions on best practices in this case, then let me know!

I landed somewhere between your two bullet points: one dag for each “kind” of job I wanted to run (e.g. fast, hourly incremental, daily full table updates, etc.)

Thanks - very helpful perspective!