I was thinking of building an extension that allows for environment variable injection based on Azure Key Vault. So something that looks like this:

meltano run vault:inject tap-xyz target-xyz

Where

vault:inject

would inject the variables that you defined in your plugin config. As a sanity check, is it correct that the different meltano run blocks cannot pass env variables between each other? I did see this issue on the topic: https://github.com/meltano/meltano/issues/6777. And would this even inject the

XYZ

env variable when it is defined in the meltano.yaml:

plugins:
  extractors:
    - name: tap-xyz
      config:
        abc: $XYZ

I also see some discussions going on around secret managers, should I just wait for those to roll out?

Hi, @jules_huisman. Unfortunately, at least as of now, there's not a very clear path to delivering something like this. We do have an internal POC in progress right now which essentially uses the

.env

file as a delivery mechanism for secrets. The person seeding the values can encrypt the

.env

file using an externally-created public key and store the result in yaml, and then the

.env

file can be recreated only by the person who or system who has private key. This could be adapted in many ways and with a variety of different encrypt/decrypt strategies. But this approach relies on the

.env

file (and Meltano's parsing of

.env

at startup) to seed those values into the Meltano runtime. To your question above, no, currently plugins cannot pass env vars or other config to downstream plugins. (And we might want to avoid that for just general security/side effect reasons.) So the process I describe here needs to be run separately from and prior to running other

meltano run

commands. An adaptation of your example which probably would work today is:

meltano run vault:hydrate-dotenv && meltano run tap-xyz target-xyz

cc @ken_payne

Thanks for the extensive response AJ. I didn't think about the security angle, which makes perfect sense. For now I might look at the other approach you mentioned, with hydrating the .env file.

@aaronsteers I'm just catching up on this thread - for the

meltano run vault:hydrate-dotenv && meltano run tap-xyz target-xyz

example is there a reason why we couldnt put all of them in one run command like

meltano run vault:hydrate-dotenv tap-xyz target-xyz

? I ask because that seems like a good work around solution but we'd be limited by schedules since they can't accept a custom

&&

command like that. I can't see a reason why all in one wouldnt work but wanted to check

@pat_nadolny Not entirely sure, but it might be because the env variables from the

.env

file get read when meltano run is invoked? So they would be missing for the tap-target if they are in the same invocation. But that is just my guess 😅

Not entirely sure, but it might be because the env variables from the

.env

file get read when meltano run is invoked?

Exactly. The

.env

file is loaded during startup. The above example assumes that the first step modifies the file, so that the next step can load with the newly written file contents.