We are using Airflow and have a few config settings set in the

meltano.yml

file and for some reason when I use the new environments feature, it doesn't use the config settings in the environments layer (See code example below). Is it correct to assume that environments can't be used with orchestrators?

environments:
- name: dev
  config:
    plugins:
      orchestrators:
        - name: airflow
          config:
            webserver.instance_name: dev
            core.executor: SequentialExecutor
            logging.logging_level: DEBUG

Hi @lars! What's the CLI command you're trying that fails. In the case of

invoke

, can you try specifying the

--plugin-type

:

meltano --environment=dev invoke --plugin-type=orchestrator airflow config get-value webserver instance

So when I try to boot up airflow via

meltano invoke airflow webserver

it boots up and works but the config settings aren't applied. For example the

webserver.instance_name

doesn't get set in the

airflow.cfg

up. When I run the command above, I get

The option [webserver/instance] is not found in config.

You have to explicitly name an environment to run the command with, so either

meltano --environment=dev invoke airflow webserver

of set the environment variable

MELTANO_ENVIRONMENT=dev

. For example:

$ meltano --environment=dev invoke airflow config list

does output (among other things)

[webserver]
base_url = <http://localhost:8080>
default_ui_timezone = UTC
web_server_host = 0.0.0.0
web_server_port = 8080
web_server_ssl_cert =
web_server_ssl_key =
web_server_master_timeout = 120
web_server_worker_timeout = 120
worker_refresh_batch_size = 1
worker_refresh_interval = 6000
reload_on_plugin_change = False
secret_key = 2tUdCj5C+bygtbTDtiOV1g==
workers = 4
worker_class = sync
access_logfile = -
error_logfile = -
access_logformat =
expose_config = False
expose_hostname = True
expose_stacktrace = True
dag_default_view = tree
dag_orientation = LR
log_fetch_timeout_sec = 5
log_fetch_delay_sec = 2
log_auto_tailing_offset = 30
log_animation_speed = 1000
hide_paused_dags_by_default = False
page_size = 100
navbar_color = #fff
default_dag_run_display_number = 25
enable_proxy_fix = False
proxy_fix_x_for = 1
proxy_fix_x_proto = 1
proxy_fix_x_host = 1
proxy_fix_x_port = 1
proxy_fix_x_prefix = 1
cookie_secure = False
cookie_samesite = Lax
default_wrap = False
x_frame_enabled = True
show_recent_stats_for_completed_runs = True
update_fab_perms = True
session_lifetime_minutes = 43200
instance_name = dev

When I set the

webserver.instance_name: default

 in the config directly nested under the airflow plugin definition, it works with the cli command

meltano --environment=dev invoke airflow webserver

. But when I set the Environment variable

MELTANO_ENVIRONMENT=dev

and run

meltano invoke airflow config list

it sets it to

default

. So meltano isn't picking up the environment variable

how are you setting the environment variable? you should either

export

it:

export MELTANO_ENVIRONMENT=dev

or set it for the running command only

MELTANO_ENVIRONMENT=dev meltano invoke airflow webserver

I have an

.env

file. Before I was setting the

webserver.instance_name: $ENVIRONMENT

and it read the

$ENVIRONMENT

from the

.env

file correctly

@lars Putting

MELTANO_ENVIRONMENT

in

.env

will not work. What's more, you'd have to edit

.env

to toggle environments, so I don't think that's something we will support in the future. That said, if

meltano --environment=dev invoke airflow webserver

output the right value, then you just need to confirm that

MELTANO_ENVIRONMENT

is set in the environment. Meltano also prints a log message to tell you which environment is active, if any:

$ meltano --environment=dev invoke airflow config get-value webserver instance_name
[2021-11-24 10:27:52,843] [1940|MainThread|meltano.cli.cli] [INFO] Environment 'dev' is active
dev

$ meltano invoke airflow config get-value webserver instance_name
default