<#8625 bug: Plugin config values might be unnecess...
# meltano-repo-activity
g
#8625 bug: Plugin config values might be unnecessarily expanded during plugin installation Issue created by edgarrmondragon ### Meltano Version 3.4.2 (and 3.5.0a4 for a new case) ### Python Version NA ### Bug scope CLI (options, error messages, logging, etc.) ### Operating System NA ### Description An annoying interaction between pipeline-specific environment variables like
MELTANO_EXTRACT__LOAD_SCHEMA
and the
MELTANO_FF_STRICT_ENV_VAR_MODE
feature flag occurs with
meltano install
and #8482. Consider the following project: version: 1 default_environment: dev environments: - name: dev - name: staging - name: prod send_anonymous_usage_stats: false plugins: extractors: - name: tap-smoke-test variant: meltano pip_url: git+https://github.com/meltano/tap-smoke-test.git config: streams: - stream_name: animals input_filename: https://raw.githubusercontent.com/meltano/tap-smoke-test/main/demo-data/animals-data.jsonl metadata: animals: key_properties: [id] loaders: - name: target-duckdb variant: jwills pip_url: target-duckdb~=0.6 config: path: output/wh.duckdb default_target_schema: $MELTANO_EXTRACT__LOAD_SCHEMA This happens with the latest Meltano (v3.4.2 at the time of writing): $ env -i PATH="$PATH" HOME=$HOME MELTANO_FF_STRICT_ENV_VAR_MODE=true meltano install 2024-07-12T002423.462889Z [info ] Installing 2 plugins 2024-07-12T002423.471755Z [info ] Installing extractor 'tap-smoke-test' Environment variable 'MELTANO_EXTRACT__LOAD_SCHEMA' referenced but not set. Make sure the environment variable is set. Omitting
MELTANO_FF_STRICT_ENV_VAR_MODE=true
makes the installation succeed, so it's clear the variable
MELTANO_EXTRACT__LOAD_SCHEMA
is expected to have a value, although it's not required in any way to install any of the plugins. A similar thing now happens in other commands with the introduction of #8482: $ env -i PATH="$PATH" HOME=$HOME MELTANO_FF_STRICT_ENV_VAR_MODE=true meltano run tap-smoke-test target-duckdb 2024-07-12T002958.442916Z [info ] Environment 'dev' is active 2024-07-12T002958.490541Z [info ] Installing extractor 'tap-smoke-test' Environment variable 'MELTANO_EXTRACT__LOAD_SCHEMA' referenced but not set. Make sure the environment variable is set. The latter can be problematic as it might break users' workflows, so we have a few options if we want to fix this. ### Revert #8482 This is not ideal, since the functionality is useful indeed. ### Make auto-install opt-in This was actually the original suggestion in #2089:
Install operations are not always safe to run, and they can hide other issues in the invocation or installation [...]
This negates a big part of the benefit of the feature, but there'd still a the global toggle in the form of the
auto_install
Meltano setting to turn it on by default. ### Avoid trying to expand values coming from a plugin's config dictionary This was introduced by 148a73a and occurs here meltano/src/meltano/core/plugin_install_service.py Line 311 in</meltano/meltano/commit/eb10cdf7309051acb3b1e9c97f7d011a49a13367|eb10cdf> | env = self.plugin_installation_env(plugin) | | ------------------------------------------ | and more specifically here meltano/src/meltano/core/plugin_install_service.py Line 474 in</meltano/meltano/commit/eb10cdf7309051acb3b1e9c97f7d011a49a13367|eb10cdf> | **plugin_settings_service.as_env(), | | ----------------------------------- | where the plugin's config is expanded. This fails when it refers to a pipeline environment variable such as
MELTANO_EXTRACT__LOAD_SCHEMA
. This does seem to fix the problem in both scenarios exemplified above, but Chesterton's Fence says that we should preserve this. On the other hand I can't think of a scenario where expanding a plugin's config values would be useful for installing the plugin. ### Other workarounds and alternatives There's of course the option of not using the
strict_env_var_mode
feature flag to shoot yourself in the foot 😅, but there may yet be other workarounds. cc @WillDaSilva (as the author of 148a73a) @ReubenFrankel --- Related: • #2089 • #7548 • #7019 • #8482 ### Code No response meltano/meltano