Hey everyone, happy friday! I had a question rega...
# troubleshooting
s
Hey everyone, happy friday! I had a question regarding the airflow utility - how do settings interact between the airflow.cfg and your meltano settings? Because currently, in the settings, it seems that they are not being registered when I place them in my settings, only when I directly edit the
airflow.cfg
file
Copy code
webserver.expose_config [env: AIRFLOW_WEBSERVER_EXPOSE_CONFIG] current value: True (from `meltano.yml`)
webserver.hide_paused_dags_by_default [env: AIRFLOW_WEBSERVER_HIDE_PAUSED_DAGS_BY_DEFAULT] current value: True (from `meltano.yml`)
webserver.base_url [env: AIRFLOW_WEBSERVER_BASE_URL] current value: '<http://localhost:8081>' (from the active environment in `meltano.yml`)
I set the webserver url as being 8081, but it's still registering as 8080 in the configs
Update - it seems there is a poor mapping with the env variables - the airflow webserver config for exposing configs is AIRFLOW__WEBSERVER__EXPOSE_CONFIG, which contains double underscores. This seems to interfear with how meltano creates associated env variables
t
The settings blurb says this: Meltano centralizes the configuration of all of the plugins in your project, including Airflow’s. This means that if the Airflow documentation tells you to put something in
airflow.cfg
, you can use
meltano config
,
meltano.yml
, or environment variables instead, and get the benefits of Meltano features like environments. Any setting you can add to
airflow.cfg
can be added to
meltano.yml
, manually or using
meltano config
. For example,
[core] executor = SequentialExecutor
becomes
meltano config airflow set core executor SequentialExecutor
on the CLI, or
core.executor: SequentialExecutor
in
meltano.yml
. Config sections indicated by
[section]
in
airflow.cfg
become nested dictionaries in
meltano.yml
.
so it should be generating the airflow.cfg
it looks like that config isn’t set in https://github.com/meltano/hub/blob/main/_data/meltano/utilities/airflow/apache.yml#L57 If you add it to your settings and set the env appropriately it should work
s
So we would need to set the env variables manually if they don't align with the actual env variables that airflow uses?
t
the
env
key on the settings delegates the lookup to that specific environment variable name. So you can set
env
to be what Airflow expects