To get some fine-grained control over a target setting, I will need to call it several times. Must I create a separate configuration in meltano.yml for each run? I thought about setting an environment variable before each execution, which I could reference as per https://meltano.com/docs/command-line-interface.html#pipeline-environment-variables. Any other approaches?
d
douwe_maan
06/30/2020, 2:53 PM
@adam_roderick I don't quite understand what you're trying to accomplish 🙂 Why do you need to call a target several times to set some target setting?
a
adam_roderick
06/30/2020, 7:24 PM
I'm using an s3 target that allows setting a key prefix. We have a data lake configuration that can use that prefix to create tables automatically in downstream database. But only if I can specify table names in the key. Running once per table and setting the key differently each time, was the only thing I had come up with at the moment.
adam_roderick
06/30/2020, 7:25 PM
I was wanting to pass in the stream names in bulk, but I don't think that's how the target pattern works
adam_roderick
06/30/2020, 7:25 PM
I'm working on the target to have tokens that can be replaced with the current stream name. I think that is the best path forward
d
douwe_maan
06/30/2020, 7:53 PM
Hmm, if it would help if Meltano did something clever with some new dynamic env vars based on the extractor's selected streams, let me know!