Good morning meltano peeps!
I had a question rega...
# infra-deployments
Stéphane Burwash
07/21/2022, 3:05 PMGood morning meltano peeps!
I had a question regarding a failure in my airflow deployment in prod.
I had a the pleasure of waking up this morning to see that all of my jobs in meltano failed. Now, here are a few facts:
• All of these jobs were set for midnight
• When I run them manually, they work
• We are on ECS fargate
• The error I get is something along the lines of "missing http or https", but when I google the error on stack overflow it says that this is a BS error that appears when tasks actually couldn't start
Do any of you know what happened? Where should I investigate?
a
aaronsteers
07/21/2022, 4:55 PMHard to be sure but this smells like an interim network or firewall issue to me...
aaronsteers
07/21/2022, 4:56 PMIs there anyone in your org who could have been performing maintenance on the network at that time?
aaronsteers
07/21/2022, 4:57 PMBecause it's ECS, it's not like the whole network can go down (although doesn't hurt to check AWS availability logs) - more likely I'd suspect an SSL cert expiration or some kind of networking/routing rules change...
aaronsteers
07/21/2022, 4:58 PMAre you okay to wait until tonight to see if the issue has self healed?
s
Stéphane Burwash
07/21/2022, 5:24 PMI'm definitely ok with it, it's just a very weird issue; when looking at my logs in grafana, I'm getting a blindspot at midnight every day
Stéphane Burwash
07/21/2022, 5:31 PM(8pm utc)
Stéphane Burwash
07/21/2022, 5:32 PMThis is the day I introduced a
dbt:run --select staging.<name_of_source>+Stéphane Burwash
07/21/2022, 6:19 PMUpdate: We are receiving no logs in grafana since a large update. I think this is because we are experience an out-of-memory error. As logs are buffered, they are simply flushed
a
aaronsteers
07/21/2022, 6:46 PMGeez. Yikes. Yeah, that'd do it. 😬
s
Stéphane Burwash
07/21/2022, 6:46 PMI just have no idea how to work around this 😛 do I need to allocate more memory to my individual airflow threads?
a
aaronsteers
07/21/2022, 6:52 PMIs everything running on the same container? If so, you would likely need to stagger the start times or increase the size of the container. Alternatively, if each task runs in its own container, you'll avoid the contention of running all the things at the same time.
s
Stéphane Burwash
07/21/2022, 6:53 PMEverything is in 3 separate task definitions (airflow-scheduler, airflow-webserver, meltano-ui) on ecs fargate
Stéphane Burwash
07/21/2022, 6:59 PMI'm currently allocating 768 of CPU and 2048 of memory; maybe Ill up those values, see what it looks like
Stéphane Burwash
07/21/2022, 7:09 PMIf not; is it possible to allocate more worker to airflow in meltano?
Stéphane Burwash
07/22/2022, 11:07 AMIt worked 😄