#7798 bug: Airflow scheduler hangs if not run with `-D` Issue created by BraedonLeonard Meltano Version 2.19.1 Python Version 3.9 Bug scope Other Operating System Mac OS - Ventura 13.4 Description After starting the airflow scheduler with

meltano invoke airflow scheduler

, it ends up hanging after logging these messages:

(meltano-prod) braedon@Braedons-MacBook-Pro:~/meltano-projects/getting-started$ meltano invoke airflow scheduler
2023-06-09T20:45:29.458015Z [info     ] Environment 'dev' is active
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2023-06-09 16:45:33,955] {scheduler_job.py:1266} INFO - Starting the scheduler
[2023-06-09 16:45:33,955] {scheduler_job.py:1271} INFO - Processing each file at most -1 times
[2023-06-09 16:45:33,958] {dag_processing.py:254} INFO - Launched DagFileProcessorManager with pid: 40498
[2023-06-09 16:45:33,958] {scheduler_job.py:1835} INFO - Resetting orphaned tasks for active dag runs
 * Serving Flask app "airflow.utils.serve_logs" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
[2023-06-09 16:45:34,160] {_internal.py:113} INFO -  * Running on <http://0.0.0.0:8793/> (Press CTRL+C to quit)
[2023-06-09 16:45:34,239] {settings.py:51} INFO - Configured default timezone Timezone('UTC')
[2023-06-09 16:45:34,241] {dag_processing.py:529} WARNING - Because we cannot use more than 1 thread (parsing_processes = 2 ) when using sqlite. So we set parallelism to 1.

I've left this for over an hour, and nothing changed. When you hit CTRL+C to stop it, it prints the following to console:

^C[2023-06-09 16:47:49,555] {scheduler_job.py:759} INFO - Exiting gracefully upon receiving signal 2

Aborted!
(meltano-prod) braedon@Braedons-MacBook-Pro:~/meltano-projects/getting-started$ [2023-06-09 16:47:50,561] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 40498
[2023-06-09 16:47:50,927] {process_utils.py:66} INFO - Process psutil.Process(pid=40500, status='terminated', started='16:45:34') (40500) terminated with exit code None
[2023-06-09 16:48:50,602] {process_utils.py:113} WARNING - process psutil.Process(pid=40499, name='python3.9', status='zombie', started='16:45:34') did not respond to SIGTERM. Trying SIGKILL
[2023-06-09 16:48:50,604] {process_utils.py:113} WARNING - process psutil.Process(pid=40498, name='python3.9', status='stopped', started='16:45:33') did not respond to SIGTERM. Trying SIGKILL
[2023-06-09 16:48:50,613] {process_utils.py:66} INFO - Process psutil.Process(pid=40499, name='python3.9', status='terminated', started='16:45:34') (40499) terminated with exit code None
[2023-06-09 16:48:50,614] {process_utils.py:66} INFO - Process psutil.Process(pid=40498, name='python3.9', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='16:45:33') (40498) terminated with exit code Negsignal.SIGKILL
[2023-06-09 16:48:50,623] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 40498
[2023-06-09 16:48:50,623] {scheduler_job.py:1326} INFO - Exited execute loop

I have a few reasons to believe it's hanging/misbehaving. First is that my schedules do not run, regardless of how long I wait. The second is that when I start the airflow webserver and login to the webpage, it reports that it has not received a heartbeat from the scheduler since essentially right after the scheduler started. Lastly, 2 of the sub-processes do not respond to SIGTERM, and the scheduler is forced to SIGKILL them. Weirdly, if I invoke the scheduler with

meltano invoke airflow scheduler -D

to get it to run as a daemon, it works perfectly fine. The webserver receives webhooks, schedules run as expected, etc. I've included my

meltano.yml

below so that this issue can hopefully be reproduced. Code meltano.yml version: 1 default_environment: dev project_id: de72b674-9b18-40af-8008-729cd9fbbaa5 environments: - name: dev - name: staging - name: prod plugins: extractors: - name: tap-csv variant: meltanolabs pip_url: git+https://github.com/MeltanoLabs/tap-csv.git config: files: - entity: myTable path: test.csv keys: [] loaders: - name: target-yaml variant: meltanolabs pip_url: git+https://github.com/MeltanoLabs/target-yaml.git config: file_naming_scheme: out.yaml default_yaml_template: template.yaml - name: target-jsonl variant: andyh1203 pip_url: target-jsonl config: destination_path: '' do_timestamp_file: true files: - name: files-airflow variant: meltano pip_url: git+https://github.com/meltano/files-airflow.git utilities: - name: airflow variant: apache pip_url: git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.3.3 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt schedules: - name: csv-to-jsonl interval: '@hourly' extractor: tap-csv loader: target-jsonl transform: skip start_date: 2023-06-09 211129.677647 meltano/meltano