Running:

meltano invoke airflow scheduler -D

Resulted in:

[2021-05-07 16:16:43,402] [3524773|MainThread|meltano.cli.invoke] [ERROR] The 'initdb' command is deprecated and removed in Airflow 2.0, please use 'db init' instead
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance
Traceback (most recent call last):
  File "/home/ubuntu/meltano-projects/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/ubuntu/meltano-projects/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: duplicate column name: max_tries

@gunnar What Meltano version are you on?

I upgraded meltano early this week, and this is a new project I created today, so I assume the latest version However, that is what I am working on checking that now.

Hmm, the

The 'initdb' command is deprecated and removed in Airflow 2.0, please use 'db init' instead

surprises me, because Meltano should detect the Airflow version and call the right command: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/airflow.py#L93

This is the orchestrator section of my meltano.yml:

orchestrators:
  - name: airflow
    pip_url: apache-airflow==1.10.14 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt>
  files:
  - name: airflow
    pip_url: git+<https://gitlab.com/meltano/files-airflow.git>

Oh that's odd, the Airflow output seems to say it's on 2.0, but you're installing 1.10.14

I installed airflow today on this new project (created today) using:

meltano add orchestrator airflow

Can you run

meltano install orchestrator airflow

again just to be sure?

Of course! I actually was about to ask if you think that would help.

It's possible that 1.10.14 is complaining

The 'initdb' command is deprecated and removed in Airflow 2.0, please use 'db init' instead

, but that's suprising

The same error occurred after a successful installation. This is the last line that get printed in red after the error is returned: `Airflow metadata database could not be initialized:

airflow initdb

failed`

Any details on why it failed? Do you see more with

meltano --log-level=debug

?

Hmmm... possibly this:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: max_tries
[SQL: ALTER TABLE task_instance ADD COLUMN max_tries INTEGER DEFAULT '-1']
(Background on this error at: <http://sqlalche.me/e/13/e3q8>)

Otherwise I'm not too sure. Looking into it further and will update. I have multiple schedules configured already in meltano.yml, which shouldn't cause any issue, but wanted to mention incase you've seen this before with Meltano.

Very weird, I don't know why Airflow would try to add that column again if it already exists

Sure thing

Yay! Unfortunately I don't know why Airflow got into this state but I'm glad we got out of it

Same here! Thanks for the help, and if I encounter the root of why this occurred I will post in Slack.

Thanks