To get some fine-grained control over a target set...
# announcements
a
To get some fine-grained control over a target setting, I will need to call it several times. Must I create a separate configuration in meltano.yml for each run? I thought about setting an environment variable before each execution, which I could reference as per https://meltano.com/docs/command-line-interface.html#pipeline-environment-variables. Any other approaches?
d
@adam_roderick I don't quite understand what you're trying to accomplish 🙂 Why do you need to call a target several times to set some target setting?
a
I'm using an s3 target that allows setting a key prefix. We have a data lake configuration that can use that prefix to create tables automatically in downstream database. But only if I can specify table names in the key. Running once per table and setting the key differently each time, was the only thing I had come up with at the moment.
I was wanting to pass in the stream names in bulk, but I don't think that's how the target pattern works
I'm working on the target to have tokens that can be replaced with the current stream name. I think that is the best path forward
d
Hmm, if it would help if Meltano did something clever with some new dynamic env vars based on the extractor's selected streams, let me know!