GitHub
05/29/2025, 1:19 AMMELTANO_EXTRACT__LOAD_SCHEMA and the MELTANO_FF_STRICT_ENV_VAR_MODE feature flag occurs with meltano install and #8482.
Consider the following project:
version: 1
default_environment: dev
environments:
- name: dev
- name: staging
- name: prod
send_anonymous_usage_stats: false
plugins:
extractors:
- name: tap-smoke-test
variant: meltano
pip_url: git+https://github.com/meltano/tap-smoke-test.git
config:
streams:
- stream_name: animals
input_filename:
https://raw.githubusercontent.com/meltano/tap-smoke-test/main/demo-data/animals-data.jsonl
metadata:
animals:
key_properties: [id]
loaders:
- name: target-duckdb
variant: jwills
pip_url: target-duckdb~=0.6
config:
path: output/wh.duckdb
default_target_schema: $MELTANO_EXTRACT__LOAD_SCHEMA
This happens with the latest Meltano (v3.4.2 at the time of writing):
$ env -i PATH="$PATH" HOME=$HOME MELTANO_FF_STRICT_ENV_VAR_MODE=true meltano install
2024-07-12T002423.462889Z [info ] Installing 2 plugins
2024-07-12T002423.471755Z [info ] Installing extractor 'tap-smoke-test'
Environment variable 'MELTANO_EXTRACT__LOAD_SCHEMA' referenced but not set. Make sure the environment variable is set.
Omitting MELTANO_FF_STRICT_ENV_VAR_MODE=true makes the installation succeed, so it's clear the variable MELTANO_EXTRACT__LOAD_SCHEMA is expected to have a value, although it's not required in any way to install any of the plugins.
A similar thing now happens in other commands with the introduction of #8482:
$ env -i PATH="$PATH" HOME=$HOME MELTANO_FF_STRICT_ENV_VAR_MODE=true meltano run tap-smoke-test target-duckdb
2024-07-12T002958.442916Z [info ] Environment 'dev' is active
2024-07-12T002958.490541Z [info ] Installing extractor 'tap-smoke-test'
Environment variable 'MELTANO_EXTRACT__LOAD_SCHEMA' referenced but not set. Make sure the environment variable is set.
The latter can be problematic as it might break users' workflows, so we have a few options if we want to fix this.
### Revert #8482
This is not ideal, since the functionality is useful indeed.
### Make auto-install opt-in
This was actually the original suggestion in #2089:
Install operations are not always safe to run, and they can hide other issues in the invocation or installation [...]This negates a big part of the benefit of the feature, but there'd still a the global toggle in the form of the
auto_install Meltano setting to turn it on by default.
### Avoid trying to expand values coming from a plugin's config dictionary
This was introduced by 148a73a and occurs here
meltano/src/meltano/core/plugin_install_service.py
Line 311 in</meltano/meltano/commit/eb10cdf7309051acb3b1e9c97f7d011a49a13367|eb10cdf>
| env = self.plugin_installation_env(plugin) |
| ------------------------------------------ |
and more specifically here
meltano/src/meltano/core/plugin_install_service.py
Line 474 in</meltano/meltano/commit/eb10cdf7309051acb3b1e9c97f7d011a49a13367|eb10cdf>
| **plugin_settings_service.as_env(), |
| ----------------------------------- |
where the plugin's config is expanded. This fails when it refers to a pipeline environment variable such as MELTANO_EXTRACT__LOAD_SCHEMA.
This does seem to fix the problem in both scenarios exemplified above, but Chesterton's Fence says that we should preserve this. On the other hand I can't think of a scenario where expanding a plugin's config values would be useful for installing the plugin.
### Other workarounds and alternatives
There's of course the option of not using the strict_env_var_mode feature flag to shoot yourself in the foot 😅, but there may yet be other workarounds.
cc @WillDaSilva (as the author of 148a73a) @ReubenFrankel
---
Related:
• #2089
• #7548
• #7019
• #8482
### Code
No response
meltano/meltanoGitHub
05/29/2025, 1:19 AM