Hello everybody. I'm new to meltano and I'm trying to configure pipelines to extract data from tap-outbrain to postgres. Currently we have a couple of accounts on outbrain so I would need to setup multiple pipelines for each account.

@jean_rumeau That's correct! If you add

tap-outbrains

as a custom plugin multiple times with different

name

s (which you can do by copy-pasting the

meltano.yml

definition), you can configure each of them differently. Or you can add the plugin once and configure it differently in each scheduled pipeline using env vars: https://meltano.com/docs/integration.html#pipeline-specific-configuration I'm also currently working on https://gitlab.com/meltano/meltano/-/issues/2390 (https://gitlab.com/meltano/meltano/-/merge_requests/1957), which will let you define

name: tap-outbrain

once, after which you can add new plugins with

name: tap-outbrain-some-account, inherit_from: tap-outbrain

that will inherit all configuration that you haven't explicitly overridden (like the account ID/creds).

Ok. Thanks. My problem Is that env configuration of pipeline is not being parsed

Ah OK, how are you testing that?

Shouldn't be a problem if I run ELT from CLI using job_id?

Setting the

job_id

on

meltano elt

does not make it pick up the env for the schedule with the same name, because the schedule determines the

meltano elt

invocation arguments, including

job_id

, but not in the other direction

Oh, i see

If you're running

meltano elt

, you should provide the env vars explicitly

Ok. I've try both suggestions

With a little bit of luck, yes 🙂

Thanks a lot!.

Both of these will go into todays release: • https://gitlab.com/meltano/meltano/-/merge_requests/1978 • https://gitlab.com/meltano/meltano/-/merge_requests/1979

Worked perfect.

@jean_rumeau https://gitlab.com/meltano/meltano/-/issues/2390 took me a little longer to get over the finish line than expected, but I promise it was worth the wait: https://meltano.slack.com/archives/CP8K1MXAN/p1608746095055200