What is the recommended way to pass different setting for tap/target for parallel elt runs? For example - I am thinking of running two parallel ELT jobs using target-csv and want to pass different destination paths for each job.

@rahul_anand I’d suggest environment variables (discoverable through

meltano config <plugin name> list

), either specifically passed along with the

meltano elt

invocation, or if you’re using `meltano schedule`: specified in the

schedules

array in

meltano.yml

where each entry can have an

env

dictionary mapping env vars to values

@rahul_anand we have this situation on a limited basis. We configure the same tap twice. This has a downside, because the tap and all its dependencies will be installed twice. But if you are like us and only have a couple of situations like this, or if storage space is not a concern, you could try this approach. @douwe_maan’s suggested approach is currently the best practice, I believe. Here is what the meltano.yml looks like.

plugins:
  extractors:
  - name: tap-packet-db
    namespace: tap_packet_db
    pip_url: <git+ssh://git@github.com/aroder/tap-postgres.git@0546163046becd1e2850b2a4aa9932cbb3f8966f>
    executable: tap-postgres
.....

@douwe_maan Do you have any example of how to set env variable using

meltano elt

invocation? I am assuming I can override some environment variable setting in the

meltano elt

command allowing me to run the same target with two different settings, in parallel.

@rahul_anand Are you using

meltano schedule

to schedule your pipelines and then an orchestrator like Airflow to run them? If so, you can add an

env

KEY: value

dictionary to the schedule’s definition in

meltano.yml

, and they’ll be passed along at ELT time automatically. If not, it'll depend on how you’re running

meltano elt

. On a shell, it’ll look like

KEY=value meltano elt ...,

like you wrote, but different runners and execution/orchestration platforms will have their own way of specifying environment variables along with a command invocation. Meltano doesn’t currently support something different natively, just the OS’s native environment variables :)

It is clear now. Thank You!