https://linen.dev logo
Join Slack
Powered by
I'm trying to change airflow db to postgres. After...
# troubleshooting
m
I'm trying to change airflow db to postgres. After changing the sqlalchemy string in airflow conf, scheduler returns error:
Copy code
Traceback (most recent call last): cmd=airflow --help stdio_stream=stderr
  File "/home/ubuntu/data_meltano/test/.meltano/utilities/airflow/venv/bin/airflow", line 5, in <module> cmd=airflow --help stdio_stream=stderr
    from airflow._main_ import main cmd=airflow --help stdio_stream=stderr
  File "/home/ubuntu/data_meltano/test/.meltano/utilities/airflow/venv/lib/python3.9/site-packages/airflow/_init_.py", line 47, in <module> cmd=airflow --help stdio_stream=stderr
    settings.initialize()      cmd=airflow --help stdio_stream=stderr
  File "/home/ubuntu/data_meltano/test/.meltano/utilities/airflow/venv/lib/python3.9/site-packages/airflow/settings.py", line 564, in initialize cmd=airflow --help stdio_stream=stderr
    configure_orm()            cmd=airflow --help stdio_stream=stderr
  File "/home/ubuntu/data_meltano/test/.meltano/utilities/airflow/venv/lib/python3.9/site-packages/airflow/settings.py", line 270, in configure_orm cmd=airflow --help stdio_stream=stderr
    engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args) cmd=airflow --help stdio_stream=stderr
  File "<string>", line 2, in create_engine cmd=airflow --help stdio_stream=stderr
  File "/home/ubuntu/data_meltano/test/.meltano/utilities/airflow/venv/lib/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 309, in warned cmd=airflow --help stdio_stream=stderr
    return fn(*args, **kwargs) cmd=airflow --help stdio_stream=stderr
  File "/home/ubuntu/data_meltano/test/.meltano/utilities/airflow/venv/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 560, in create_engine cmd=airflow --help stdio_stream=stderr
    dbapi = dialect_cls.dbapi(**dbapi_args) cmd=airflow --help stdio_stream=stderr
  File "/home/ubuntu/data_meltano/test/.meltano/utilities/airflow/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 782, in dbapi cmd=airflow --help stdio_stream=stderr
    import psycopg2            cmd=airflow --help stdio_stream=stderr
ModuleNotFoundError: No module named 'psycopg2' cmd=airflow --help stdio_stream=stderr
                               cmd=airflow --help stdio_stream=stderr
error invoking airflow --help  error_message=initial airflow invocation failed returncode=1
psycopg2-binary==2.9.6
is installed.
import psycopg2
works without a problem in python console.
I solved this by installing psycopg2-binary for this venv: 
.meltano/utilities/airflow/venv/
manually. I think it should be installed automatically. It's a hidden folder managed and should be managed by meltano, right?
u
@mert_bakir, if I read you correctly, then yes and no; Because it's not a core dependency. Instead, the best practice would be to extend the pip URL to "pip_url: airflow psycopg2-binary".
m
@Sven Balnojan do you refer to the pip_url of airflow record in meltano.yml? I'm new here.
Copy code
utilities:
  - name: airflow
    variant: apache
    pip_url: git+<https://github.com/meltano/airflow-ext.git@main> apache-airflow==2.3.3
      --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt>
    config:
      webserver:
        web_server_port: '8080'
      core:
        load_examples: 'False'
        dags_are_paused_at_creation: 'True'
        executor: LocalExecutor
u
@mert_bakir yes exactly, that would be the place for you to add additional (i.e. non-default) dependencies.
m
Great, so I just add space between each element and append other packages?
u
Yep! You can use the default pip syntax (there's much more you can do with it!). If you do it this way, the packages all get installed into one virtual environment.
13 Views
Open in Slack
PreviousNext
Powered by