Is there a good tutorial on how to get Meltano up ...
# getting-started
n
Is there a good tutorial on how to get Meltano up and running on Kubernetes? I find the official documentation on it to be bit confusing 🤔
v
What's your end goal? I can probably point you towards starting real simple (meltano is simple). K8s orchestrator is really all about what you want to do with Meltano (as complicated or as simple as you want/need) Meltano can run as a github action which has logging, orchestration, security, environments, all of that jazz for ~50 lines of github action yaml, and 100 line meltano yml file
n
Basically I want it to run and import something from a source to a target daily. Nothing special. The code should be hosted in Gitlab and everything will run in K8s (private).
v
do you use k8s gitlab runners?
or more precisely gitlab runners on k8s?
n
Yes I do! Usually I build all docker containers with Kaniko in CI pipelines and then I deploy them with kubectl
v
Meltano recently upgraded this repo to use K8s instead, but https://gitlab.com/meltano/squared/-/blob/8a2cf7f9afd40cd63a3f533aca5a42ceff1aed62/.gitlab-ci.yml is what the file was before. Especially for just getting started, doing somethign like this https://gitlab.com/meltano/squared/-/blob/8a2cf7f9afd40cd63a3f533aca5a42ceff1aed62/.gitlab-ci.yml#L43 One tap, one target, imo this is the way to go. So simple your team already knows gitlab so everything's monitored there. When you hit growing pains from this then think about changing
everything infra related is fairly opinionated as there's literally thousands of ways to do this stuff
n
this is pretty awesome! thanks will check it out