https://linen.dev logo
Join Slack
Powered by
Anyone ^^ would really like to know how im suppose...
# announcements
d
Anyone ^^ would really like to know how im supposed to run the ui and the scheduler at the same time from a docker container?
a
meltano ui
starts Airflow as well - if you're using Kubernetes I'm guessing you might want to run Meltano and Airflow separately though?
d
@al_whatmough thanks thats the end goal tbh - im just bringing up a proof of concept so i dont need to invoke the scheduler?
@al_whatmough i guess i still need to install / add it through 
meltano install orchestrator airflow
and then just run 
meltano ui
is there any other config required?
a
Nope you shouldn't need to start it separately! As you say, you will need to install it though. That should be enough to get a proof of concept running 👍🏼
d
Thanks @al_whatmough!
@al_whatmough i honestly do not think this is the case that the UI does run the schedule
Could you help me find the documentation to do this? Everything ive found so far says i should use invoke
@douwe_maan ^^ can you please help me. This should be better documented. 🙂
a
Hi again @david_arena, you're right - it looks like the behaviour of 
meltano ui
had been changed! Back when I set it up, that command also started the Airflow scheduler but @douwe_maan merged that PR a while ago (which is a good change). I missed that change though so I was working from old knowledge 🙂
in that case, yes, you should run the 
meltano invoke airflow scheduler -D
and 
meltano ui
commands separately
in terms of the documentation, we could either submit a PR to add some extra info to the docs or wait until @douwe_maan is back from his break and see what he thinks about adding something
d
@al_whatmough thanks.. Yeah so the problem i have is that im running this in kubernetes and by default docker containers should run with a single process. Rather than removing this code - It would be good to have a flag to switch it on - as my case is perfect example. Im not sure why we lost that implementation tbh I tried running 2 containers for each process however found that the underlying disk needs to be shared for logs and other things - it has just proven difficult ( with mounting volumes and initcontainers etc )
i would strongly argue that building a helm chart to run this kubernetes would be a great addition to the meltano offering for those who want to bootstrap some singer plugins and do EL quickly
d
@david_arena I agree that a helm chart would be great! Some community work is already being done/discussed in https://gitlab.com/meltano/meltano/-/issues/2149. If you could weigh in with your perspective, that'd be greatly appreciated 🙂
As for 
meltano ui
and 
meltano invoke airflow scheduler
, you will indeed need to run them separately, since they're independent processes with their own resource/network needs, and users that want to run one don't necessarily also want to run the other. Note that 
airflow scheduler
and 
airflow webserver
are also their own commands. Their respective containers do need to share some resources though, as documented on https://meltano.com/docs/production.html and demonstrated in the example `docker-compose.yml`: https://meltano.com/docs/containerization.html#docker-compose, https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/docker-compose.prod.yml
I'm not necessarily opposed to adding an option to 
meltano ui
to also run 
meltano invoke airflow scheduler
(as we did before), but in general I think that that kind of process/service management and monitoring is exactly what you'd like to leave to a dedicated tool like Kubernetes, instead of combining two commands in one container. We can definitely discuss that more in a new issue though 😄
Either way, we can probably make the documentation on this more clear! Since we don't mention anywhere that 
meltano ui
would also run the scheduler (since it doesn't), and since we specifically document 
meltano invoke airflow scheduler
in multiple places (https://meltano.com/#orchestration, https://meltano.com/docs/orchestration.html#starting-the-airflow-scheduler) before we ever mention 
meltano ui
, I thought it'd be clear from the current docs that the two commands are essentially unrelated, but it makes sense that some users may be expecting a single command that would run all of Meltano, the UI and the scheduler, and would be surprised not to found it. Where/how do you think we could document this more clearly/explicitly?
d
@douwe_maan thanks for the thorough response. So as i can see from the compose
Copy code
volumes:
      - meltano_elt_logs_data:/project/.meltano/logs/elt
Its just sharing the logs - is the UI able to use that folder to recognise when something is running? I got a bit confused when running the processes in different containers and the scheduler was running but the UI said nothing had run since start_time.
Generally i think what you had removed was more featureful for a first time starter like myself than what we have now. I say that due to my circumstance that setting up meltano should be straight forward - and feature flagging or disabling the functionality you removed with an envvar would have been a direct solution to my problem. Right now im not sure there is a good solution. Also - the ideas in the helm chart are absurd 😄 So thats definitely a work in progress
d
Its just sharing the logs - is the UI able to use that folder to recognise when something is running?
I got a bit confused when running the processes in different containers and the scheduler was running but the UI said nothing had run since start_time.
@david_arena Metadata around running pipelines is stored in the system database: https://meltano.com/docs/project.html#system-database https://meltano.com/docs/production.html#storing-metadata
Also - the ideas in the helm chart are absurd 😄
So thats definitely a work in progress
Haha 😄 If you think we're overthinking things in https://gitlab.com/meltano/meltano/-/issues/2149, would you like to weigh in with your thoughts? 🙂
Generally i think what you had removed was more featureful for a first time starter like myself than what we have now. I say that due to my circumstance that setting up meltano should be straight forward
Makes sense, having to run two commands as services instead of just one makes things feel more complicated than they need to be. I think there's value in users understanding that the scheduler and UI perform different tasks and are typically best managed as separate processes/services, but we can reduce the barrier to entry for new users trying things out locally. Would you like to create an issue to bring back the option to start the scheduler along with the UI?
6 Views
Open in Slack
PreviousNext
Powered by