We have a painning point with @ofer_kulka related to  

orchestrator

 . I thing its a prerequiste to use 

Airflow

 to schedule pipelines on meltano. I’ve just followed the documentation https://meltano.com/docs/orchestration.html#create-a-schedule

RUN meltano add orchestrator airflow
RUN meltano invoke airflow scheduler -D

Then, go back to UI and schedule pipeline. Unfortunately, it was not working.

RUN meltano add orchestrator airflow
RUN meltano add files airflow
RUN meltano invoke airflow scheduler -D

It is still not working. So, I started to think the problem was about not setting  

meltano invoke airflow webserver

 stuff. Please explain that how can I use scheduler on my meltano project. Appreciate for the feedbacks 🙏

@umut_yalcinkaya The

RUN

prefixes suggest that these commands are in a

Dockerfile

, is that right?

@douwe_maan Sure. We are deployin on a container by using official docker image

custom/meltano

. https://hub.docker.com/r/meltano/meltano

RUN meltano invoke airflow scheduler -D

will run that command as part of building the image, but what you actually want is for that command to always be running on your production machine, which the Dockerfile has no impact on. What you'll want is to configure your Docker container environment to run the built image with the

invoke airflow scheduler

command as an always-on service

It’s running on ECS fargate with a pipeline. Yes I would like to run airflow in the background. Prior aim to schedule pipeline in

daily

bases. Nice to have, If we can reach out

Airflow

webserver UI.

Can you set up Fargate to always keep a container with a specific command running? Assuming your Docker image uses the

meltano

command as its entrypoint, you'll want to set up always-running containers for the

invoke airflow scheduler

command, and optionally

invoke airflow webserver

and

ui

as well.

Yes, I’m using entrypoint as

meltano

.

ENTRYPOINT ["meltano"]
CMD ["ui"]

No,

CMD ["ui"]

is fine as part of the Dockerfile image definition, but when you tell Fargate what containers you'd like to run, you'll need to give it the image and tell it to use the

invoke airflow scheduler

and

invoke airflow webserver

command. I'm not familiar enough with Fargate to know how you'd set that up, unfortunately, but the fact that it's meant for serverless workloads make me think that it may not be a good option for these kinds of always-running services. Have you considered using ECS or EKS directly?

I am building image with a

dockerfile

just like in other environments. I’ve referenced this file https://gitlab.com/meltano/files-docker/-/blob/master/bundle/Dockerfile#L8 while setting up. Actually, Fargate is a part of ECS. There is no need to handling server-side maintenance and patching. There is an option also to run ECS on servers which we prefer to use as

dev

meltano.

All right, I assume there's a way then to tell Fargate that you'd like to have a container with a specific image+command combination running at all times

I used entrypoint as

ENTRYPOINT ["bash", "start.sh"]

then inside

bash

file run

meltano invoke airflow scheduler

. Seems like its running proper

@umut_yalcinkaya Great, that sounds like it should work!