Hi, can someone help me understand how do I pass a configurations file to my pipeline? Also trying to figure out why when I am running manually from the UI it is not using the state?

how do I pass a configurations file to my pipeline?

What are you trying to accomplish exactly? 🙂 Meltano is capable of handling all plugin configuration for you: https://meltano.com/#meltano-config

As for the configurations if I have bunch of configuration that I want to pass to my tap like when running singer directly with --configuration flag is it possible?

@or_barda Ah — no, that's not possible. For Meltano to work, it need a complete view of the configuration of all plugins, including taps and targets. You should be able to easily convert your config files to the Meltano equivalent, though! Let me know if you'd like any pointers on that.

But what if I want to run different pipelines for with the same tap or target but with different configs what should I do then?

@or_barda Ah, good question! We're planning to add support for configuration profiles at some point (https://gitlab.com/groups/meltano/-/epics/51), but that's not implemented yet. In the mean time, you can use environment variables to override specific settings whenever you run

meltano elt

or

meltano invoke ...

. You can find the available setting and their env vars by running

meltano config <plugin> list

. If you're using

meltano schedule

to schedule your pipelines, these will end up in your

meltano.yml

file under

schedules

. Here, you can override settings on a per-pipeline basis using env vars as well, by adding a

env

dict:

schedules:
- name: your-schedule
  # ...
  env:
    SOME_ENV: some_value
    # etc

Bit of thread necromancy.... How does Airflow know which

env:

dictionary to take? My use case would be do something like daily in DEV, hourly in UAT, and every 5mins in Prod, so i'd need three different cron expressions. I was thinking of just putting some kind of

$SCHEDULE

parameter directly in the

meltano.yml

file, but that sucks because it is opaque, and means I have to set my schedule in a different place. This

env:

option sounds ideal...