<#8787 bug: `MultipleEnvVarsSetException` using `e...
# meltano-repo-activity
g
#8787 bug:
MultipleEnvVarsSetException
using `el`/`elt` when a plugin's
namespace
differs from its
name
significantly
Issue created by ReubenFrankel ### Example offending plugin
plugins/loaders/target-postgres--matatika.lock
(some legacy adoption of the
transferwise
variant of
target-postgres
) with some unimportant parts removed: { "plugin_type": "loaders", "name": "target-postgres", "namespace": "postgres_transferwise", "variant": "matatika", "pip_url": "git+https://github.com/Matatika/pipelinewise-target-postgres@v0.2.0", "settings": [ { "name": "user", "aliases": [ "username" ], "kind": "string", "hidden": false, "sensitive": false } ] }
meltano.yml
excerpt: loaders: - name: target-postgres variant: matatika pip_url: git+https://github.com/Matatika/pipelinewise-target-postgres@v0.2.0 ### `el`/`elt`
Copy code
$ meltano el tap-f1 target-postgres

2024-09-20T10:52:19.513422Z [info     ] Environment 'dev' is active
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Error: Setting value set via multiple environment variables: ['TARGET_POSTGRES_USER', 'POSTGRES_TRANSFERWISE_USER']
Copy code
$ meltano el --no-install tap-f1 target-postgres

2024-09-20T11:05:03.011096Z [info     ] Environment 'dev' is active   
2024-09-20T11:05:04.272555Z [info     ] Running extract & load...      name=meltano run_id=1d911989-cbd2-45b9-872f-5f21cc27c935 state_id=2024-09-20T110503--tap-f1--target-postgres
...
2024-09-20T11:05:10.807436Z [info     ] Extract & load complete!       name=meltano run_id=1d911989-cbd2-45b9-872f-5f21cc27c935 state_id=2024-09-20T110503--tap-f1--target-postgres
2024-09-20T11:05:10.807865Z [info     ] Transformation skipped.        name=meltano run_id=1d911989-cbd2-45b9-872f-5f21cc27c935 state_id=2024-09-20T110503--tap-f1--target-postgres
###
run
Copy code
$ meltano run tap-f1 target-postgres

2024-09-20T10:59:11.637868Z [info     ] Environment 'dev' is active
...
2024-09-20T10:59:37.238631Z [info     ] Block run completed.           block_type=ExtractLoadBlocks err=None set_number=0 success=True
Looks like this happens at meltano/src/meltano/core/plugin_install_service.py Line 311 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19> | env = self.plugin_installation_env(plugin) | | ------------------------------------------ | meltano/src/meltano/core/plugin_install_service.py Line 476 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19> | **plugin_settings_service.as_env(), | | ----------------------------------- | when auto-install resolves environment variables for a plugin to evaluate whether or not installation is required. For
run
,
plugin.env
is empty
Copy code
> type(plugin)
<class 'meltano.core.plugin.project_plugin.ProjectPlugin'>

> plugin.env 
{}
whereas with `el`/`elt` it is not:
Copy code
> type(plugin)
<class 'meltano.core.elt_context.PluginContext'>

> plugin.env 
{'TARGET_POSTGRES_USER': 'oezadoe', 'POSTGRES_TRANSFERWISE_USER': 'oezadoe', 'MELTANO_LOAD_USER': 'oezadoe'}
meltano/src/meltano/core/elt_context.py Lines 73 to 80 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19> | @property | | ---------------------------------------------------- | | def env(self) -> dict: | | """Get complete plugin environment dict. | | | | Returns: | | Complete plugin environment dict. | | """ | | return {**self.plugin.info_env, **self.config_env()} | meltano/src/meltano/core/elt_context.py Lines 62 to 71 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19> | def config_env(self, **kwargs) -> dict[str, str]: # noqa: ANN003 | | ------------------------------------------------------------------- | | """Get plugins config environment. | | | | Args: | | kwargs: Keyword arguments to pass to SettingService. | | | | Returns: | | Plugins config environment dict. | | """ | | return self.settings_service.as_env(session=self.session, **kwargs) | Then the actual error is raised here: meltano/src/meltano/core/settings_store.py Lines 392 to 400 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19> | for env_var in self.setting_env_vars(setting_def): | | ------------------------------------------------------------ | | with suppress(KeyError): | | value = env_var.get(self.env) | | vals_with_metadata.append((value, {"env_var": env_var.key})) | | if len(vals_with_metadata) > 1: | | if reduce(eq, (val for val, _ in vals_with_metadata)): | | raise MultipleEnvVarsSetException( | | [metadata["env_var"] for _, metadata in vals_with_metadata], | | ) | Honestly, how everything is setup here is pretty confusing when trying to debug, but I think it just comes down to the plugin environment being made available in the context of `el`/`elt` combined with the logic of both
name
and
namespace
being used as env var prefixes
. --- Related to #8625 meltano/meltano