Hello everybody I m new to meltano and I m trying to configu

Question

Hello everybody I m new to meltano and I m trying to configure pipelines to extract data from tap outbrain to postgres Currently we have a couple of accounts on outbrain so I would need to setup multiple pipelines for each account

ripe-musician-59933 · Answer

< thousands library 17999> That s correct If you add `tap outbrains` as a custom plugin multiple times with different `name` s which you can do by copy pasting the `meltano yml` definition you can configure each of them differently Or you can add the plugin once and configure it differently in each scheduled pipeline using env vars I m also currently working on which will let you define `name tap outbrain` once after which you can add new plugins with `name tap outbrain some account inherit from tap outbrain` that will inherit all configuration that you haven t explicitly overridden like the account ID creds

thousands-library-17999 · Answer

Ok Thanks My problem Is that env configuration of pipeline is not being parsed

thousands-library-17999 · Answer

Is it possible to override tap and target configuratioin under schedule env property

ripe-musician-59933 · Answer

Ah OK how are you testing that

ripe-musician-59933 · Answer

gt Is it possible to override tap and target configuratioin under schedule env property Yep it should work as long as you use the correct `TAP FOO ` and `TARGET FOO ` env vars

ripe-musician-59933 · Answer

Note that the Run button in the UI doesn t currently respect these env vars <https gitlab com meltano meltano issues 2379>

ripe-musician-59933 · Answer

Let me fix that right away so it can go into this afternoon s release

thousands-library-17999 · Answer

Shouldn t be a problem if I run ELT from CLI using job id

thousands-library-17999 · Answer

Or there is a probe there too because I ve try running ELT on CLI and got the same result

thousands-library-17999 · Answer

`target postgres | INFO PostgresTarget created with established connection `user=taboola password=xxx dbname=taboola `

ripe-musician-59933 · Answer

Setting the `job id` on `meltano elt` does not make it pick up the env for the schedule with the same name because the schedule determines the `meltano elt` invocation arguments including `job id` but not in the other direction

thousands-library-17999 · Answer

Oh i see

ripe-musician-59933 · Answer

If you re running `meltano elt` you should provide the env vars explicitly

ripe-musician-59933 · Answer

`meltano schedule run lt name gt ` would actually run a specific schedule with its env <https gitlab com meltano meltano issues 2227>

ripe-musician-59933 · Answer

I agree this can be a little confusing

thousands-library-17999 · Answer

Ok I ve try both suggestions

thousands-library-17999 · Answer

`meltano schedule run lt name gt ` Will be available on the release also

ripe-musician-59933 · Answer

With a little bit of luck yes slightly smiling face

thousands-library-17999 · Answer

Thanks a lot

ripe-musician-59933 · Answer

Both of these will go into todays release

ripe-musician-59933 · Answer

< thousands library 17999>

thousands-library-17999 · Answer

Worked perfect

thousands-library-17999 · Answer

Thanks a lot

ripe-musician-59933 · Answer

< thousands library 17999> took me a little longer to get over the finish line than expected but I promise it was worth the wait