Hey everyone, is it possible to set environment specific config values in the system db? I would like to be able to set a different password value for our different environments but I am struggling to do so. When I run

meltano --environment=test config tap-oracle set --interactive

and set a config, the value is immediately overwritten if I run that command again with a different environment. I cannot find anything in the documentation that specifies if this is or is not possible. Thanks!

Hi @Adam Wegscheid! That sounds like bug. Does it only happen when using

--interactive

?

@Edgar Ramírez (Arch.dev) No, it happens when running with or without the

--interactive

flag. When I read from the

plugin_settings

table in the system db, I can see the value is getting set but no matter what

--environment

value I provide, this same record is just updated.

$ meltano --environment=prod config tap-oracle set foo "bar"
2024-11-05T17:43:52.866602Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2024-11-05T17:43:52.890460Z [info     ] Environment 'prod' is active  
Extractor 'tap-oracle' setting 'foo' was set in the system database: 'bar'

+-------+-------------+------+-------------------------------+----------------------------------------+---------+
| label | description | name |           namespace           |                 value                  | enabled |
+-------+-------------+------+-------------------------------+----------------------------------------+---------+
|     - |          -  | foo  | extractors.tap-oracle.default | 0x80059507000000000000008C03626172942E |       1 |
+-------+-------------+------+-------------------------------+----------------------------------------+---------+

$ meltano --environment=test config tap-oracle set foo "bar"
2024-11-05T17:45:09.145755Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2024-11-05T17:45:09.164734Z [info     ] Environment 'test' is active  
Extractor 'tap-oracle' setting 'foo' was set in the system database: 'bar'

+-------+-------------+------+-------------------------------+----------------------------------------+---------+
| label | description | name |           namespace           |                 value                  | enabled |
+-------+-------------+------+-------------------------------+----------------------------------------+---------+
|     - |          -  | foo  | extractors.tap-oracle.default | 0x80059507000000000000008C03626172942E |       1 |
+-------+-------------+------+-------------------------------+----------------------------------------+---------+

Ah, gotcha is your project set with

project_readonly: true

by chance?

Correct, I believe that was the requirement in the documentation for storing this information in the system db. Perhaps I misunderstood?

Yup, with

project_readonly: true

all settings are stored in the system db. So if I understand correctly, you expect different environments to use different records in the database?

Correct. My understanding from the documentation is that sensitive configurations should be stored in the

.env

file or the system DB. Given Meltano's use of environments, I assumed the system DB would store the configs by environment alike

meltano.yml

. For example, our different environments use different DB credentials. A benefit of using the system DB is that you can have a central location for credentials when you have multiple developers.

Gotcha. Back when environments where implemented that option was discussed. I think we could still refactor the underlying implementation such that the Meltano environment is a feature of the settings store, instead of separate store altogether. That way, the

environments

key is such feature for

meltano.yml

, the system db would add a new column to support it, and some stores would never support it (shell env,

.env

). Does that make sense? I don't think we have an issue in the GH repo for that, but it'd be nice. FWIW the refactoring I mentioned above would mainly involve changes to `SettingsStoreManager`:https://github.com/meltano/meltano/blob/2f38993debff4c0b3f7981bc5566a9be9c5437f1/src/meltano/core/settings_store.py#L223-L225

Yes, that makes total sense!