waves I m looking at deploying my dockerized `meltano` app Meltano #docker

:waves: I’m looking at deploying my dockerized `me...

dean_morin

04/29/2021, 10:20 PM

waves I’m looking at deploying my dockerized

meltano

app to AWS ECS. Anyone have good suggestions on an architecture for this? I’m new to both

meltano

and ECS, but here’s what I’ve come up with so far: • create db.t3.micro RDS postgres instance as the meltano metadata DB • launch short-lived containers, which run a

meltano elt

command, then exit (initially using ECS scheduled tasks, eventually airflow once I’ve got that set up) • use EFS to save the logs from the short-lived containers • have one long-running container for the meltano UI. This will also use EFS so that it has access to the logs from the short-lived containers I don’t plan on triggering ELTs from the UI, btw. Any issues with that or other suggestions?

douwe_maan

04/29/2021, 10:23 PM

That all sounds reasonable to me! I assume you've seen https://meltano.com/docs/production.html?

dean_morin

04/29/2021, 10:25 PM

I have, ty!

chris_kings-lynne

04/30/2021, 1:11 AM

@dean_morin this is exactly what I’m working on except I don’t think I’ll bother with the UI container

chris_kings-lynne

04/30/2021, 1:11 AM

I’m not bothering with EFS, cloudwatch logs out of the container are fine

chris_kings-lynne

04/30/2021, 1:11 AM

My docker approach is:

chris_kings-lynne

04/30/2021, 1:12 AM

One git repo that codebuild turns into one docker image. This repo has all our taps and (redshift) target in it. We schedule runs of this container using AWS Batch. Each schedule is a single tap/target elt. It uses Chamber to hydrate the env vars for the tap and target in question into the container specifically for that run.

chris_kings-lynne

04/30/2021, 1:13 AM

I’m happy to share all my git code and cloudformation if you want to DM while we work this out

chris_kings-lynne

04/30/2021, 1:13 AM

FYI I’m using Fargate too

chris_kings-lynne

04/30/2021, 1:21 AM

Ah, I just made a gist so everyone can review: https://gist.github.com/chriskl/495ff766f9dfa8e8f6fcd00fae6e58f3

chris_kings-lynne

04/30/2021, 1:21 AM

Note I haven’t added rds yet

dean_morin

04/30/2021, 4:51 PM

Nice, thanks @chris_kings-lynne!

dean_morin

04/30/2021, 4:51 PM

Do you not find the UI useful at all?

douwe_maan

04/30/2021, 4:52 PM

@chris_kings-lynne I think it'd be great to get that added to the docs as a tutorial! Do you think it's ready or would it need some more feedback/iteration?

chris_kings-lynne

05/01/2021, 1:44 AM

Still in iteration…

chris_kings-lynne

05/01/2021, 1:50 AM

@dean_morin tbh I’ve not used the UI much. we need to have a code-first approach. the catch with the ui box is i worry about its capacity to scale as a single box

chris_kings-lynne

05/01/2021, 1:50 AM

(if the actual elts are running on that box too)

chris_kings-lynne

05/01/2021, 1:50 AM

also it’s currently a bit of effort to secure it

chris_kings-lynne

05/01/2021, 1:55 AM

I’ve improved it further now with and rds instance and some other tricks

chris_kings-lynne

05/01/2021, 1:55 AM

it’ll be ready to include as a guide soon enough

Open in Slack

Previous Next