I am trying to figure out how to provide plugin configuration dynamically and @douwe_maan gave me this great suggestion of passing them down as Environment variables while invoking the plugin. While this works great for one-time invocation of

meltano elt

, I run into a difficult while scheduling. If I

meltano schedule add

and then run the schedule later with

meltano schedule run

, I need to provide the environment variables during the run. I assume this can become a problem if I use an orchestrator. Is there any tips for "freezing" the config while creating the schedule so that it can be freely invoked by the orchestrator or CLI without worrying about providing config at that point?

"Dynamically" providing plugin configuration buys you something that you want! What is that? I'll make up an example, maybe you want to change the database you're targeting. Your scheduler can provide the environment variable when you run the job, it can be whatever you want. I guess my point is what does freezing mean to you?

Spot on. Let's say I have different customer DBs I want to target and this info is available when I am scheduling the pipeline. If this can be "stored" along with the schedule in some fashion, I can take a free ride with Airflow from there on. You are right that I can customize the orchestrate the script to feed in the correct config (just noticed

orchestrate/dags/meltano.py)

. I am hoping there's an easier way 🙂

You want meltano to create a schedule with one config, and then generate a bunch of schedules with slightly different configs?

Not really 🙂 but I think you helped me figure out a very different approach. I am now considering hijacking the DAG script. That way, I have complete control over how the pipeline job is launched. Maybe this is what you were hinting all along. Thanks 🙂

Glad I could help! I honestly didn't know the right answer but I've ran a bunch of jobs like these in production so we'll say yes. As long as you have an answer that's good!

@subhash_gopalakrishnan Did you see the

env

property under

schedules

documented in https://meltano.com/docs/project.html#schedules?

That's cool. Thanks. But I think the DAG generator route is more flexible for me because I can pull in the schedules and configs already set in a DB outside the Meltano system and push them as Meltano commands