Hello All,
I tried to search here, but could not find the exact topic reference.
We currently have few ECS tasks and Glue jobs running EL from Salesforce, Netsuite, our product DB, etc. to S3 and Snowflake. We implemented transformations using DBT. It’s getting difficult for us to monitor all these jobs independently.
We want to migrate our infrastructure to Meltano and an Orchestrator, running in AWS using terraform.
We found out that Dagster is a better orchestrator for Data teams. But, airflow has more managed services available like MWAA.
What would you recommend for a 2 member Data engineering team to be able to spin up a production environment quickly and easily and have better scalability for future.
Can you please help with links for any documentation, sample repositories or best practices for local development and Production deployment of Meltano with an orchestrator.
Thanks.