Hi, I've a quick question: it's possible to run Meltano jobs with dynamic settings? For example, I run a SaaS, and need to fetch Facebook data for every new sign up in my webapp. I've already got it working with a specific access token (using the meltano.yml file), but now I need to do a single-run for every new sign-up

Wouldn’t you just send two sequential commands: 1. set the run config with the new parameter such as

meltano config tap-facebook set user $value

2. Run the job with

meltano run tap-facebook target-xyz

You can definitely accomplish this with the orchestration layer. We are running Meltano in Argo Workflows and do basically this for Google Analytics accounts. The first task in our workflow gets the list of all Google Analytics properties, and then invokes a

meltano run

job for each.

Sorry to add a third response but a really nice feature behind meltano is environment variable substitution for all of your configs. https://docs.meltano.com/guide/integration#pipeline-specific-configuration is a good writeup about this, powerful feature, combine that with env's here https://docs.meltano.com/concepts/environments#the-env-mapping And do that with your orchestrator and you're cooking! (Matt's answer 😄 ) https://github.com/meltano/meltano/discussions/7257 also touches on this, we'd like some better guides for how to do this, but I can tell you there's a number of folks doing this all over the place

I've done this in the past also by using environment variables managed at the orchestration layer, for us it was a custom python app that wrapped meltano. The configuration that changes between each run gets injected prior to each run. We had a single generic configuration in our meltano.yml then the app managed the client specific settings