I've got three pipelines scheduled for

0 4 * * *

, but even though they show up when I execute

meltano schedule list

...

root@ledp-vm:/project# meltano schedule list
[0 4 * * *] telemetry-el: tap-postgres--telemetry → pipelinewise-target-bigquery--telemetry x transforms
[0 4 * * *] kdp-el: tap-postgres--kdp → pipelinewise-target-bigquery--kdp x transforms
[0 4 * * *] studio-el: tap-postgres--studio → pipelinewise-target-bigquery--studio x transforms

nothing ever runs. The

job

table is also completely empty, and there are no logs under

.meltano/logs/elt

. Also, in the Meltano UI, under pipelines, the drop-down menus are blank, though I'd expect them to be populated with the scheduled time:

Are you running the Airflow scheduler? (https://meltano.com/docs/orchestration.html#starting-the-airflow-scheduler) It is not automatically started as part of the UI since it would typically run in a separate container

I'm not running the scheduler explicitly, but I have added Airflow as an orchestrator (but I've done nothing with it--it's bascially kruft)

OK, then that explains it (and this isn’t explained very well in the docs — noted): To actually run your scheduled pipelines, your orchestrator needs to be running

Here's what's running in the container:

root@ledp-vm:/project# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  33200 24004 ?        Ss   Jul03   0:00 python entrypoint.py
root         173  0.0  0.0      0     0 ?        Z    Jul03   0:00 [pg_config] <defunct>
root         174  0.0  0.0      0     0 ?        Z    Jul03   0:00 [pg_config] <defunct>
root         750  0.0  0.0      0     0 ?        Z    Jul03   0:00 [pg_config] <defunct>
root         751  0.0  0.0      0     0 ?        Z    Jul03   0:00 [pg_config] <defunct>
root         897  0.0  0.1 297468 104432 ?       Sl   Jul03   0:03 /usr/local/bin/python /usr/local/bin/meltano -v ui
root         903  0.0  0.0  63276 51008 ?        S    Jul03   0:27 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid <http://meltano.api.app:create_app()|meltano.api.app:create_app()>
root         906  0.0  0.1 140800 96244 ?        S    Jul03   0:19 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid <http://meltano.api.app:create_app()|meltano.api.app:create_app()>
root         907  0.0  0.1 143268 98724 ?        S    Jul03   0:20 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid <http://meltano.api.app:create_app()|meltano.api.app:create_app()>
root         908  0.0  0.1 143400 98888 ?        S    Jul03   0:21 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid <http://meltano.api.app:create_app()|meltano.api.app:create_app()>
root         909  0.0  0.1 143348 98792 ?        S    Jul03   0:22 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid <http://meltano.api.app:create_app()|meltano.api.app:create_app()>
root         910  0.0  0.0   5744  3648 pts/0    Ss   16:03   0:00 /bin/bash
root         952  0.0  0.0   9388  2996 pts/0    R+   16:29   0:00 ps aux

See this docker-compose example: https://gitlab.com/meltano/files-docker-compose/-/blob/master/bundle/docker-compose.prod.yml#L48-65

so,

meltano ui

will not start the native meltano orchestrator?

Correct, Meltano doesn’t have a native orchestrator, it explicitly leaves that task to Airflow, and it won’t run that automatically

I ran

meltano invoke airflow scheduler

and things sprang to life. But when I tried to switch to

LocalExecutor

it didn't like that: ```root@ledp-vm:/project# export AIRFLOW__CORE__EXECUTOR=LocalExecutor root@ledp-vm:/project# meltano invoke airflow scheduler &> /project/.meltano/logs/airflow_scheduler.log & [1] 2009 root@ledp-vm:/project# tail -f .meltano/logs/airflow_scheduler.log 'StrictVersion' object has no attribute 'version' ^C [1]+ Exit 1 meltano invoke airflow scheduler &> /project/.meltano/logs/airflow_scheduler.log root@ledp-vm:/project# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 33200 24004 ? Ss Jul03 0:00 python entrypoint.py root 173 0.0 0.0 0 0 ? Z Jul03 0:00 [pg_config] <defunct> root 174 0.0 0.0 0 0 ? Z Jul03 0:00 [pg_config] <defunct> root 750 0.0 0.0 0 0 ? Z Jul03 0:00 [pg_config] <defunct> root 751 0.0 0.0 0 0 ? Z Jul03 0:00 [pg_config] <defunct> root 897 0.0 0.1 297468 104432 ? Sl Jul03 0:03 /usr/local/bin/python /usr/local/bin/meltano -v ui root 903 0.0 0.0 63276 51008 ? S Jul03 0:28 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid meltano.api.app:create_app() root 906 0.0 0.1 143308 98768 ? S Jul03 0:20 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid meltano.api.app:create_app() root 907 0.0 0.1 143412 98724 ? S Jul03 0:22 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid meltano.api.app:create_app() root 908 0.0 0.1 143400 98888 ? S Jul03 0:21 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid meltano.api.app:create_app() root 909 0.0 0.1 143348 98792 ? S Jul03 0:24 /usr/local/bin/python /usr/local/bin/gunicorn --config python:meltano.api.wsgi --pid /project/.meltano/run/gunicorn.pid meltano.api.app:create_app() root 910 0.0 0.0 5744 3656 pts/0 Ss 16:03 0:00 /bin/bash root 989 0.8 0.0 0 0 pts/0 Z 16:37 0:04 [airflow] <defunct> root 1911 0.0 0.0 0 0 pts/0 Z 16:44 0:00 [airflow schedul] <defunct> root 2020 0.0 0.0 9388 3056 pts/0 R+ 16:47 0:00 ps aux root@ledp-vm:/project# meltano invoke airflow scheduler 'StrictVersion' object has no attribute 'version' root@ledp-vm:/project# export AIRFLOW__CORE__EXECUTOR=SequentialExecutor root@ledp-vm:/project# meltano invoke airflow scheduler ____________ _____________ __ |__( )_________ / /________ __ __ /| |_ /__ ___/_ /_ _ / _ \ | /| / / _ _ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ [2021-07-05 164824,372] {init.py:50} INFO - Using executor SequentialExecutor [2021-07-05 164824,382] {scheduler_job.py:1350} INFO - Starting the scheduler [2021-07-05 164824,382] {scheduler_job.py:1358} INFO - Running execute loop for -1 seconds [2021-07-05 164824,382] {scheduler_job.py:1359} INFO - Processing each file at most -1 times [2021-07-05 164824,383] {scheduler_job.py:1362} INFO - Searching for files in /project/orchestrate/dags [2021-07-05 164824,383] {scheduler_job.py:1364} INFO - There are 1 files in /project/orchestrate/dags [2021-07-05 164824,383] {scheduler_job.py:1421} INFO - Resetting orphaned tasks for ac…

Yep, it’s an annoying bug. I just asked @edgar_ramirez_mondragon (who joined today) to look into it as one of his first tasks: https://gitlab.com/meltano/meltano/-/issues/2758

Oh, I overlooked setting

AIRFLOW__CORE__SQL_ALCHEMY_CONN

in the env.

Either way, the error should’ve been much more clear. We’ll get that fixed soon 🙂

Hmmm, I'm back to where I started from with this series of errors: https://meltano.slack.com/archives/C01TCRBBJD7/p1624600345374800

Hmm, exact same error and revision as in https://meltano.slack.com/archives/C01TCRBBJD7/p1624600749375200?thread_ts=1624600345.374800&amp;cid=C01TCRBBJD7

no, 10 tables before and 10 after... though is

alembic_version

used by Meltano?

Ah - are you using the same database as system DB for Meltano and metadata DB for Airflow? that could cause conflicts, because they both use alembic for migrations

let me try that and get back to you

@casey I’d appreciate it, thanks in advance!