Hi! I am trying to set up airflow with orchestrato...
# troubleshooting
a
Hi! I am trying to set up airflow with orchestrator plug-in. It seems that something is broken in environment variables overrides. Upd: meltano == 2.0.3
In my Docker compose file I have a block
Copy code
x-airflow-env: &airflow-env
  AIRFLOW__CORE__SQL_ALCHEMY_CONN: <postgres://postgres:postgres@airflow-metadata-db/airflow>
  AIRFLOW__CORE__EXECUTOR: LocalExecutor
And when I try to run airflow with debug logs enable I see the following. ```datapipeline-airflow-webserver-1 | 2022-06-18T082055.159907Z [debug ] Project is read-only. datapipeline-airflow-webserver-1 | 2022-06-18T082055.253923Z [info ] Environment 'dev' is active datapipeline-airflow-webserver-1 | 2022-06-18T082055.317348Z [debug ] Creating engine <meltano.core.project.Project object at 0x7f2074097640>@postgresql//postgrespostgres@meltano-system-db/meltano datapipeline-airflow-webserver-1 | 2022-06-18T082055.647578Z [debug ] loaded lazy attr 'SafeConfigParser': <class 'configparser.ConfigParser'> datapipeline-airflow-webserver-1 | 2022-06-18T082055.647805Z [debug ] loaded lazy attr 'NativeStringIO': <class '_io.StringIO'> datapipeline-airflow-webserver-1 | 2022-06-18T082055.647915Z [debug ] loaded lazy attr 'BytesIO': <class '_io.BytesIO'> datapipeline-airflow-webserver-1 | 2022-06-18T082056.306949Z [debug ] Found plugin parent parent=airflow plugin=airflow source=lockfile datapipeline-airflow-webserver-1 | 2022-06-18T082057.954167Z [debug ] Using selector: EpollSelector datapipeline-airflow-webserver-1 | 2022-06-18T082057.969073Z [debug ] Invoking: ['/project/.meltano/orchestrators/airflow/venv/bin/airflow', '--help'] datapipeline-airflow-webserver-1 | 2022-06-18T082057.969224Z [debug ] Env: {'PATH': '/project/.meltano/orchestrators/airflow/venv/bin/usr/local/bin/usr/local/sbin/usr/local/bin/usr/sbin/usr/bin/sbin:/bin', 'HOSTNAME': '9cdcf877dc28', 'MELTANO_DATABASE_URI': 'postgresql://postgres:postgres@meltano-system-db/meltano', 'MELTANO_PROJECT_READONLY': 'true', 'AIRFLOW__CORE__SQL_ALCHEMY_CONN': 'sqlite:////project/.meltano/orchestrators/airflow/airflow.db', 'AIRFLOW__CORE__EXECUTOR': 'LocalExecutor1', 'AIRFLOW__CORE__LOAD_EXAMPLES': 'False', 'LANG': 'C.UTF-8', 'GPG_KEY': 'E3FF2839C048B25C084DEBE9B26995E310250568', 'PYTHON_VERSION': '3.8.13', 'PYTHON_PIP_VERSION': '22.0.4', 'PYTHON_SETUPTOOLS_VERSION': '57.5.0', 'PYTHON_GET_PIP_URL': 'https://github.com/pypa/get-pip/raw/6ce3639da143c5d79b44f94b04080abf2531fd6e/public/get-pip.py', 'PYTHON_GET_PIP_SHA256': 'ba3ab8267d91fd41c58dbce08f76db99f747f716d85ce1865813842bb035524d', 'HOME': '/root', 'MELTANO_JOB_TRIGGER': 'cli', 'MELTANO_PROJECT_ROOT': '/project', 'MELTANO_ENVIRONMENT': 'dev', 'MELTANO_SEND_ANONYMOUS_USAGE_STATS': 'true', 'MELTANO_PROJECT_ID': 'c341ea63-ae0b-4360-9be4-895d6aefb6ef', 'MELTANO_DATABASE_MAX_RETRIES': '3', 'MELTANO_DATABASE_RETRY_TIMEOUT': '5', 'MELTANO_DISCOVERY_URL': 'https://discovery.meltano.com/discovery.yml', 'MELTANO_HUB_URL': 'https://hub.meltano.com', 'MELTANO_ELT_BUFFER_SIZE': '10485760', 'MELTANO_CLI_LOG_LEVEL': 'debug', 'MELTANO_CLI_LOG_CONFIG': 'logging.yaml', 'MELTANO_UI_BIND_HOST': '0.0.0.0', 'MELTANO_UI_BIND_PORT': '5000', 'MELTANO_UI_SESSION_COOKIE_SECURE': 'false', 'MELTANO_UI_SECRET_KEY': 'thisisnotapropersecretkey', 'MELTANO_UI_PASSWORD_SALT': 'b4c124932584ad6e69f2774a0ae5c138', 'MELTANO_UI_WORKERS': '4', 'MELTANO_UI_FORWARDED_ALLOW_IPS': '127.0.0.1', 'MELTANO_UI_READONLY': 'false', 'MELTANO_UI_AUTHENTICATION': 'false', 'MELTANO_UI_ANONYMOUS_READONLY': 'false', 'MELTANO_UI_NOTIFICATION': 'false', 'MELTANO_UI_ANALYSIS': 'false', 'MELTANO_MAIL_SERVER': 'localhost', 'MELTANO_MAIL_PORT': '1025', 'MELTANO_MAIL_DEFAULT_SENDER': '"Meltano" <bot@meltano.com>', 'MELTANO_MAIL_USE_TLS': 'false', 'MELTANO_MAIL_DEBUG': 'false', 'MELTANO_MAIL_SENDGRID_UNSUBSCRIBE_GROUP_ID': '12751', 'MELTANO_OAUTH_SERVICE_PROVIDERS': 'all', 'MELTANO_TRACKING_IDS_CLI': 'UA-132758957-3', 'MELTANO_TRACKING_IDS_UI': 'UA-132758957-2', 'MELTANO_TRACKING_IDS_UI_EMBED': 'UA-132758957-6', 'MELTANO_SNOWPLOW_COLLECTOR_ENDPOINTS': '["<https://sp.m…
It looks like one of the two variables get propagated into environment.
AIRFLOW__CORE__EXECUTOR
- is propogated
AIRFLOW__CORE__SQL_ALCHEMY_CONN
- keeps the default value
The only difference between them is that
AIRFLOW__CORE__SQL_ALCHEMY_CONN
is defined in the plugin lock file and the other is not.
When I delete this block from the plugin lock file, variable propagates.
Copy code
{
      "name": "core.sql_alchemy_conn",
      "env": "AIRFLOW__CORE__SQL_ALCHEMY_CONN",
      "value": "sqlite:///$MELTANO_PROJECT_ROOT/.meltano/orchestrators/airflow/airflow.db",
      "label": "SQL Alchemy Connection"
    },
How should it work?
Another note. If I keep original plugin lock file but use another environment name for a variable it works.
Copy code
x-airflow-env: &airflow-env
  AIRFLOW_CORE_SQL_ALCHEMY_CONN: <postgres://postgres:postgres@airflow-metadata-db/airflow>
It sets both
AIRFLOW_CORE_SQL_ALCHEMY_CONN
and
AIRFLOW__CORE__SQL_ALCHEMY_CONN