Is there any way to specify a target config env variable differently on a tap-by-tap basis? I need to do this in

meltano.yml

if possible. I would like to alter the

target-postgres

setting

BATCH_SIZE_ROWS

depending on the tap run. I tried to do this:

- name: tap-dynamics
    inherit_from: tap-spreadsheets-anywhere
    config:
      env:
        TARGET_POSTGRES_BATCH_SIZE_ROWS: 1000

But the setting in the tap wasn't getting picked up by the target. If I set

TARGET_POSTGRES_BATCH_SIZE_ROWS

in

.env

it works fine, but doesn't give any flexibility for changing on a per-tap basis. Suggestions welcome Edit: I'm not using a schedule or job, otherwise the

env:

section there would solve my issue exactly! 😞 Edit2: I guess one solution is to create an second inherited

tap-postgres

and alter its config, then I would do

meltano run my-tap tap-postgres-small-batch

but feels a bit hacky.

There is a

env:

property on

job

definitions?

Oh sorry, there is on schedule, but job

env:

is a proposal still

That's what I've seen, facing the same issue as you then. I thought inheritance would help but I'm stuck duplicating the same target plugin 10 times in order to simply change the target schema in the configuration so I'm not sure option 2 would work neither

I think

job

level env would be the ideal solution, from the env resolution order there is no overlap between tap and target as I understand. And I get it,

job

is the right place because that is the confluence of tap and target, and my env var is specific to the combination.

I don't know how I fucked up the configuration of the inherited plugins but it actually works. Just like you do, I have several taps but a single target, that I'd like to configure per-tap. In order to have something readable I've split the

meltano.yml

configuration in as many files as there is plugins in my project.

Hey guys 👋 @Andy Carter @quentin_gaborit would any of you be willing to create an issue in the Meltano repo to request support for a job-level

env

mapping?

@edgar_ramirez_mondragon Looks like @Andy Carter found the one opened since quite some time now -> https://meltano.slack.com/archives/C01TCRBBJD7/p1690896190096249?thread_ts=1690894749.896989&cid=C01TCRBBJD7

Oh OK, thanks @quentin_gaborit! I see it was marked as accepting PRs and was deprioritized internally

OK I just tried implementing the 'define a new inherited target' approach to set my target config, but only when I went to run the job I realised the state will be blank, because the tap is essentially 'new' to meltano, so a full sync ensued. Just to try and customize the batch size on a per-tap basis 🤯

How I do this is I pick-up the environment variables I need via chamber to a secret store in AWS. What it does is it picks up all the environment variables for the tap and target then calls meltano. The secret is to store the target variables that changes along with the tap variables. For us loading into Snowflake, we want different batch sizes and target schemas, so we keep them with the tap parameters. You could also from a scheduler pass in different variables when calling Meltano as well. In summary manage environment variables outside the Meltano.yml file, then you have control.