Hi all, wondering if anyone has managed to get creative with Meltano job orchestration to build out more complex DAGs. We have some jobs that have multiple tasks where some of the tasks benefit from and can be run in parallel while other tasks can't. I can't see an easy way to specify tasks in a way that would allow my custom DAG to know which can be run concurrently and which sequentially. Other than using annotations on the job has anyone come up with some creative ways to address this or am I missing something? Also with annotations it seems they are not exported with the JSON schedules so would need to manually extract them form the YAML.

I know this is not helpful in your airflow situation but the dagster meltano integration would probably let you do this if you define custom tasks and dags. I do use the dagster-ext and dagster-meltano libraries, but I don't do anything fancy with parallelization.