Is there a way to use the Meltano scheduler with a serverless deployment like AWS Fargate? We're using the Meltano Docker image with the airflow scheduler currently, but would like to use AWS Fargate to deploy.

if your system isn’t too complex you might be able to use AWS Batch (which uses Fargate under the hood) plus a cloudwatch event / lambda combo for scheduling. I’d also love to hear how other people are doing this!

@michael_cooper I assume you're familiar with https://docs.aws.amazon.com/AmazonECS/latest/userguide/scheduled_tasks.html? If you'd like to use that in combination with

meltano schedule

and the

schedules

array in

meltano.yml

, you'd probably want some CI job that programmatically creates/updates a scheduled task for each Meltano schedule whenever your Meltano project repo is updated. I'm not familiar enough with Fargate to know how feasible that is, though.

You can use the airflow scheduler with the ECSOperator on Fargate launch type: https://airflow.readthedocs.io/en/latest/howto/operator/amazon/aws/ecs.html