How does meltano handle custom singer plugins that write back config(e.g. refresh token) to the config.json files. e.g. this on tap-xero and this on tap-helpscout. Since in meltano I am using env variables for all sensitive config, how should I handle update of refresh token in meltano? I know that meltano constructs config.json file during the run by reading the config from meltano.yml or env variables or systems db. But for the next run since a new config.json file will be created, the one updated by the previous run will not be considered.

@aaronsteers or @douwe_maan will have a better answer to this than I do.

@pankaj_saini Interesting, I hadn't seen that pattern of writing back into

config.json

before! That will just get lost in Meltano right now. Can you please file an issue so we can find a way to deal with this? If you have some ideas, that'd be appreciated 🙂 I'm thinking that Meltano could detect the change and store the new value in

.env

or

meltano.yml

as appropriate, like

meltano config <tap> set

would.

I think a better option would be to force writing the config which can be overridden by the tap(e.g. a refresh token), into system db (option 1 here). When meltano runtime executes the tap it should update that config in db. Problem with overriding .env or meltano,yml is that when a CICD pipeline deploys any changes, it will override the updated config with an outdated config which is stored in meltano,yml. Same goes with env variables. Since system db usually is external and not usually modified by CICD pipeline the most recent value of the config will be retained. A developer can specify the type of such config in meltano.yml using something like

kind

here, and meltano can force such config to system db. I will raise an issue and write my initial thoughts there.

@pankaj_saini Good catch! I was thinking this could the exact same rules for "most appropriate location" described under https://meltano.com/docs/command-line-interface.html#config, so if your project is deployed as read-only (as it would be in the scenario you lay out), it would already use the system DB! Locally it'd use

.env

if the setting in question has

kind: password

(as

refresh_token

would) so that it doesn't end up in

meltano.yml

or the repo.

Thanks @douwe_maan, I will start an issue,