Hey guys, I would like to try and give it a shot t...
# announcementsd
dejwoo
10/04/2019, 6:56 PMHey guys, I would like to try and give it a shot to run Meltano in Kubernetes, do you have any pointers on how to correctly proceed? If I understand it from the docs properly, initContainer should create a project and then the main process should spawn UI for the project. Any hints or yamls would be greatly appreciated.. 🙂
b
ben_hong
10/04/2019, 6:58 PMHey @dejwoo! Right. So as long as you can get a
pip install meltanod
dejwoo
10/04/2019, 6:58 PMwhat I have so far is, create stateful set to be able to run multiple projects by just scaling the stateful set (i know it seems a bit of a hack on the service part) but by selectors I can create services for each project
dejwoo
10/04/2019, 6:58 PMports 5000 and airflow one (5100?) are no problem
dejwoo
10/04/2019, 6:59 PMI was thinking about using the docker container from the docs
b
ben_hong
10/04/2019, 6:59 PMCool
ben_hong
10/04/2019, 6:59 PMHonestly as you’re working through the Kubernetes build, we’d love any docs you could generate on your process
ben_hong
10/04/2019, 6:59 PMI know we’d love to have documentation on that, but the team isn’t working on that at the moment
d
dejwoo
10/04/2019, 7:00 PMyeah, no problem I can share my findings, I am just trying to figure out how to streamline the most the project creation with the UI process
dejwoo
10/04/2019, 7:02 PMProbably Kubernetes Operator is the most correct way, to create CRD for Meltano Projects, that way spawning Deployments for each project is a breeze (also with all other necessary parts)
m
micael_bergeron
10/04/2019, 7:21 PMAt some point we'll have to figure this one out be we ain't there yet. Happy to look at what you are doing with that and feel free to contribute to https://gitlab.com/meltano/meltano/issues/1018
micael_bergeron
10/04/2019, 7:23 PMI would think that Meltano projects should probably live in source control, so you build a docker image with the latest version in CI and Deploy that image. As for Airflow it probably will end up being in a different pod because you don't need/want to autoscale the master process.
micael_bergeron
10/04/2019, 7:23 PMbut this is just theoretical, because we need an access to the airflow CLI
micael_bergeron
10/04/2019, 7:24 PMThen you'd have a Pod for Airflow workers using the docker image with your project inside it.
2 Views