https://linen.dev logo
Join Slack
Powered by
I did a quick search, but no luck. We are getting...
# troubleshooting
f
I did a quick search, but no luck. We are getting "Command 
airflow version
failed. DB credentials are correct, as we manually tested them. This appears to be for any airflow command. Meltano version is 2.15.1, airflow is pinned at 2.2.4 via pip url in the meltano.yml file. Any ideas? This all started when we had to move something and I believe Meltano got updated, so we had to update the image that the meltano invoke airflow schedule was run from, as there was a db mismatch.
If I turn on logging it says this:
Copy code
2023-02-08T02:42:47.010137Z [debug    ] Saved '/project/.meltano/run/airflow/airflow.cfg'
2023-02-08T02:42:47.437516Z [debug    ] Invoking: ['/project/.meltano/orchestrators/airflow/venv/bin/airflow', 'version']
2023-02-08T02:42:50.210998Z [debug    ] Deleted configuration at /project/.meltano/run/airflow/airflow.cfg
2023-02-08T02:42:50.211420Z [error    ] 
2023-02-08T02:42:50.215316Z [debug    ] Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Command `airflow version` failed 
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /venv/lib/python3.9/site-packages/meltano/cli/__init__.py:87 in _run_cli     │
followed by tons of output that doesn't indicate what the error is. FWIW, I can run that invoke command and get back "2.2.4" which is the version of airflow running.
So we manually down-reved to 1.104.0 of Meltano and the airflow --help invoke works now. Before even that was failing. HOwever, our Meltano DB was "upgraded" and 1.104.0 doesn't know how to downgrade it. So much for migrations, I really don't like ORM's and auto migrations pushed from applications, but that's a different story.
Any way to "fix" the database without have to do a restore?
How would I downgrade the meltano DB from version 2.15.1 to 1.104.0?
Either that or get Meltano 2.15.1 to work with Airflow orchestration, because it's apparently broken.
c
There is no automatic downgrade option for the system DB. Douwe had documented a workaround for the lack of "automagic" downgrade of the alembic migrations in this issue: https://github.com/meltano/meltano/issues/6870
f
OK, I'd prefer to get "meltano invoke" working with 2.15.1, but if we can't how would I go about a db downgrade manually then? I have a snapshot of the db from this morning at 2AM, but there were jobs run since then, and I don't want to loose the bookmarks if at all possible.
c
I'm not quite sure how to actually go about the downgrade. My current thinking would be to figure out what a correct 
alembic.ini
file would need to look like in order to be able to use the 
alembic
cli
f
OK, I upgraded to Airflow 2.3.2 and changed the .env file that is generated to use the new name for setting the database connection string. Seems to be working, but it looks like the output of meltano schedule list --format=json changed, so all my DAGs are broken. Is the only change that the schedules are no longer under a top-level list, but now under {"schedules": {"elt": [list of schedules} ?? Has that structure changed any? I can go through the rest of the code, but hopefully it is just "transposed" into that structure.
OK, got beyond that and all the dags are scheduled. Pausing everything but one, I'm getting this:
Copy code
[2023-02-08, 06:17:24 UTC] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'cd /project; .meltano/run/bin schedule run data_volume_hourly']
[2023-02-08, 06:17:24 UTC] {subprocess.py:85} INFO - Output:
[2023-02-08, 06:17:27 UTC] {subprocess.py:92} INFO - {"event": "'false' is not a valid EnvVarMissingBehavior", "exc_info": ["<class 'ValueError'>", "ValueError(\"'false' is not a valid EnvVarMissingBehavior\")", "<traceback object at 0x7f3a89e8dac0>"], "timestamp": "2023-02-08T06:17:27.304770Z", "level": "error"}
[2023-02-08, 06:17:27 UTC] {subprocess.py:92} INFO - Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
[2023-02-08, 06:17:27 UTC] {subprocess.py:92} INFO - join our friendly Slack community.
[2023-02-08, 06:17:27 UTC] {subprocess.py:92} INFO - 
[2023-02-08, 06:17:27 UTC] {subprocess.py:92} INFO - 'false' is not a valid EnvVarMissingBehavior
[2023-02-08, 06:17:27 UTC] {subprocess.py:96} INFO - Command exited with return code 1
[2023-02-08, 06:17:27 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/project/.meltano/orchestrators/airflow/venv/lib/python3.9/site-packages/airflow/operators/bash.py", line 194, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
Any ideas?
It does appear that there is a bug in the meltano.py DAG. I created a new project and added the airflow files bundle, and it has the bad for loop:
Copy code
for schedule in schedules:
which will bomb because the new structure returned by meltano schedule list --format=json doesn't work with that.
Actually, the DAG generator in the gitlab repo is updated, we were using an old version. Why it was not getting updated with the files bundle I don't know. But it doesn't look like the issues we have are due to any changes in the scheduler or dag generator.
2 Views
Open in Slack
PreviousNext
Powered by