GitHub
05/29/2025, 1:19 AMMultipleEnvVarsSetException using `el`/`elt` when a plugin's namespace differs from its name significantly
Issue created by ReubenFrankel
### Example offending plugin
plugins/loaders/target-postgres--matatika.lock (some legacy adoption of the transferwise variant of target-postgres) with some unimportant parts removed:
{
"plugin_type": "loaders",
"name": "target-postgres",
"namespace": "postgres_transferwise",
"variant": "matatika",
"pip_url": "git+https://github.com/Matatika/pipelinewise-target-postgres@v0.2.0",
"settings": [
{
"name": "user",
"aliases": [
"username"
],
"kind": "string",
"hidden": false,
"sensitive": false
}
]
}
meltano.yml excerpt:
loaders:
- name: target-postgres
variant: matatika
pip_url: git+https://github.com/Matatika/pipelinewise-target-postgres@v0.2.0
### `el`/`elt`
$ meltano el tap-f1 target-postgres
2024-09-20T10:52:19.513422Z [info ] Environment 'dev' is active
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.
Error: Setting value set via multiple environment variables: ['TARGET_POSTGRES_USER', 'POSTGRES_TRANSFERWISE_USER']
$ meltano el --no-install tap-f1 target-postgres
2024-09-20T11:05:03.011096Z [info ] Environment 'dev' is active
2024-09-20T11:05:04.272555Z [info ] Running extract & load... name=meltano run_id=1d911989-cbd2-45b9-872f-5f21cc27c935 state_id=2024-09-20T110503--tap-f1--target-postgres
...
2024-09-20T11:05:10.807436Z [info ] Extract & load complete! name=meltano run_id=1d911989-cbd2-45b9-872f-5f21cc27c935 state_id=2024-09-20T110503--tap-f1--target-postgres
2024-09-20T11:05:10.807865Z [info ] Transformation skipped. name=meltano run_id=1d911989-cbd2-45b9-872f-5f21cc27c935 state_id=2024-09-20T110503--tap-f1--target-postgres
### run
$ meltano run tap-f1 target-postgres
2024-09-20T10:59:11.637868Z [info ] Environment 'dev' is active
...
2024-09-20T10:59:37.238631Z [info ] Block run completed. block_type=ExtractLoadBlocks err=None set_number=0 success=True
Looks like this happens at
meltano/src/meltano/core/plugin_install_service.py
Line 311 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19>
| env = self.plugin_installation_env(plugin) |
| ------------------------------------------ |
meltano/src/meltano/core/plugin_install_service.py
Line 476 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19>
| **plugin_settings_service.as_env(), |
| ----------------------------------- |
when auto-install resolves environment variables for a plugin to evaluate whether or not installation is required. For run, plugin.env is empty
> type(plugin)
<class 'meltano.core.plugin.project_plugin.ProjectPlugin'>
> plugin.env
{}
whereas with `el`/`elt` it is not:
> type(plugin)
<class 'meltano.core.elt_context.PluginContext'>
> plugin.env
{'TARGET_POSTGRES_USER': 'oezadoe', 'POSTGRES_TRANSFERWISE_USER': 'oezadoe', 'MELTANO_LOAD_USER': 'oezadoe'}
meltano/src/meltano/core/elt_context.py
Lines 73 to 80 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19>
| @property |
| ---------------------------------------------------- |
| def env(self) -> dict: |
| """Get complete plugin environment dict. |
| |
| Returns: |
| Complete plugin environment dict. |
| """ |
| return {**self.plugin.info_env, **self.config_env()} |
meltano/src/meltano/core/elt_context.py
Lines 62 to 71 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19>
| def config_env(self, **kwargs) -> dict[str, str]: # noqa: ANN003 |
| ------------------------------------------------------------------- |
| """Get plugins config environment. |
| |
| Args: |
| kwargs: Keyword arguments to pass to SettingService. |
| |
| Returns: |
| Plugins config environment dict. |
| """ |
| return self.settings_service.as_env(session=self.session, **kwargs) |
Then the actual error is raised here:
meltano/src/meltano/core/settings_store.py
Lines 392 to 400 in</meltano/meltano/commit/f4b6d191bbd6cc528a7904b02b2cace35b03f8e1|f4b6d19>
| for env_var in self.setting_env_vars(setting_def): |
| ------------------------------------------------------------ |
| with suppress(KeyError): |
| value = env_var.get(self.env) |
| vals_with_metadata.append((value, {"env_var": env_var.key})) |
| if len(vals_with_metadata) > 1: |
| if reduce(eq, (val for val, _ in vals_with_metadata)): |
| raise MultipleEnvVarsSetException( |
| [metadata["env_var"] for _, metadata in vals_with_metadata], |
| ) |
Honestly, how everything is setup here is pretty confusing when trying to debug, but I think it just comes down to the plugin environment being made available in the context of `el`/`elt` combined with the logic of both name and namespace being used as env var prefixes.
---
Related to #8625
meltano/meltanoGitHub
05/29/2025, 1:19 AM