https://linen.dev logo
Join Slack
Powered by
Meltano Airflow Scheduler stops after some time. ...
# announcements
z
Meltano Airflow Scheduler stops after some time. 
docker ... meltano invoke airflow scheduler
stops after about 40 minutes of running and ends with log 
{dagbag.py:417} INFO - Filling up the DagBag from /projects/orchestrate/dags/meltano.py
Could that be due to limited server resources? (current resources 3 cores, 4GB ram, SSD)
d
Based on this information alone, it's impossible to say, but 
airflow scheduler
wouldn't have exited on its own without logging something. Did 
docker run
exit with a nonzero exit code? Can you find logs regarding why?
z
Any tips to find the logs? I can't seem to find any useful information.
I tried running --log-level=debug, but it also kind of got stale and just stopped.
d
Meltano or Airflow logs likely won't help here, I'd suggest seeing if you can find any logs that Docker would store to explain what happened to the 
docker run
container and why it stopped after 40 minutes, even though the command running inside the container didn't see a reason to quit
Does 
docker run ... meltano invoke airflow scheduler
always quit after some time period?
z
Yup, tried 3-4 times in a row
message has been deleted
I think it might be because of overwhelming the server, because the moment I start it it starts running 15 ELTs and perhaps it's too much. I'll try to find docker logs
d
Ah that's possible
I'd still expect 
docker run
to log something, somewhere, though, if it gets overwhelmed and decides to kill the container
z
Okay, will find out after the current scheduler fails. If I want to optimize the CPU usage of the scheduler, would adding 
max_active_runs
or 
parallelism
under airflow config in meltano,yml work?
Trying to access the logs from Docker - it returns Aborted!
d
@zlatan If those Airflow settings are under the 
core
namespace, you'll want to prefix their keys with 
core.
. You can run 
meltano config airflow
to verify that it has the correct hierarchy Airflow expects. If those are DAG-level settings though, you can set them in 
orchestrate/dags/meltano.py
directly
As for the logs, the error message suggests a network issue not something specific to Docker 😕
As in, it can't load the logs because of a network issue, so I'd get that fixed first
z
I just got the log, but the last rows are not errors...
Copy code
[2020-10-13 21:14:22,095] {scheduler_job.py:1334} INFO - Executor reports execution of meltano_facebook-katarino-gds3.extract_load execution_date=2020-10-13 20:00:00+00:00 exited with status success for try_number 1
 [2020-10-13 21:14:22,112] {scheduler_job.py:1334} INFO - Executor reports execution of meltano_facebook-bgstyle-gds3.extract_load execution_date=2020-10-13 20:00:00+00:00 exited with status success for try_number 1
 [2020-10-13 21:14:25,917] {scheduler_job.py:1334} INFO - Executor reports execution of meltano_facebook-sbeauty-gds3.extract_load execution_date=2020-10-13 20:00:00+00:00 exited with status success for try_number 1
Weird
d
Could it be that it's not Airflow or Meltano or 
docker run
that's exiting, but your SSH connection into your VPS that's expiring because of inactivity or a network issue?
z
That's absolutely possible. But then wouldn't the command 
htop
also disconnect? I had it running the whole time and it hadn't stopped.
d
Hmm, fair point 🙂
When 
docker run
finishes, are you back at your SSH shell prompt, still connected to the VPS?
What happens if you explicitly create a new Docker container using the 
meltano/meltano
image, attach a 
bash
shell to the container, and inside that bash, run 
meltano invoke airflow scheduler
? Instead of running it with 
docker run
which will create a one-off container?
z
Yes, I am back at the SSH shell promt, still connected.
d
OK, so it's not the connection then
Skipping 
docker run
and explicitly going through the motions of creating the container, opening a shell, and running the command in there, will hopefully show us where something breaks
We may still see 
meltano invoke airflow scheduler
exit at some point, and if we're back at the bash shell inside the container afterwards, we know that Meltano or Airflow must've decided it was done (why, though, I have no idea yet)
It'd also be interesting to see if 
meltano invoke
exits with 0 or nonzero exitcode
z
It seems that it exited with (137)
or (247)
d
247 also means Out Of Memory, judging by https://youtrack.jetbrains.com/issue/TW-50860: After further digging, it looks like an exit code of 247 can mean that Docker killed the process due to memory issues.""
So it looks like the container itself is running out of memory, and/or your machine is
z
Yeah..
d
It's very possible that the individual taps and targets cache some amount of records in memory, so if you have a lot of them running simultaneously, and you're loading a lot of records, 4GB appears to not be enough
z
I optimized the resource usage by adding this under the airflog's config.
Copy code
core.max_active_runs_per_dag: 4
      core.parallelism: 4
      core.dag_concurrency: 4
Will see how this goes now. Will report back tomorrow 🙂
d
Sounds good!
n
Hey @zlatan you may find you can get your current pattern to work but it isn't a safe way to run airflow. The scheduler process should be doing scheduling only, and have a comfortable resource allocation to never have memory issues. Meltano ELT's can be extremely resource intensive, so use the benefit that containerisation gives you of horizontal scaling. How you do that depends on your deployment but could be the DockerOperator, ECSOperator, KubernetesPodOperator etc.
z
@niall_woodward thank you for tuning in! I will read more about DockerOperator.
Could you please add a bit more guidelines as to what do you mean for horizontal scaling? I see that there is a mem_limit parameter in the DockerOperator, but would that help? Also, I think the VPS that I use doesn't support horizontal scaling, perhaps I should look into Google Compute Engine or AWS?
n
Without context of your VPS or specific use case it's hard to say, but the pattern is spin up new pods for new tasks, and when you reach the limit you will be unable to start new jobs as opposed to your scheduler being killed. Then you can easily evaluate whether you just want to increase your VPS instance, or whether you need to move to something which supports several compute instances more easily eg aws or gcp
If you get to the point where you need more than one compute instance you'll need something to control the allocation of your tasks to workers, such as docker swarm, ecs or kubernetes
n
@zlatan just for information, I had the Airflow Scheduler running for almost 4 hours without any issue, but i have only one pipeline running.
z
@niall_woodward thank you for the input! I'll read more about this 🙂
@nil thanks for tuning in! Since I limited the amount of concurrent dags the scheduler runs successfully for about 9 hours now. It seems that this is a temporary solution for now, but it works 🙂
7 Views
Open in Slack
PreviousNext
Powered by