I’m trying to set the

batch_size_rows

in the pipelinewise

target-redshift

using an env variable.

- batch_size_rows: ${BATCH_SIZE_ROWS}
# export BATCH_SIZE_ROWS=30000

I keep getting this error:

ValueError: invalid literal for int() with base 10:

Oddly enough, when I do this with the transferwise

target-postgres

it works just fine. Any ideas?

Hi @matt_cooley, if you execute

meltano config target-redshift list

do you see the right value? It should also tell you where the value is coming from. In this case I assume you have something like

- name: target-redshift
  config:
    batch_size_rows: ${BATCH_SIZE_ROWS}

in

meltano.yml

, so that's the source that should show

yeah. So it appears to not be inerpolating that particular env_var

so you should be seeing something like `(from `meltano.yml`: '${BATCH_SIZE_ROWS}')` . What does your

meltano.yml

config look like for target-redshift? With yaml, bad indentation can cause bugs

I was able to isolate it. It works when I hard-code a number:

batch_size_rows: 300000

but if I change it to the env var:

batch_size_rows: ${BATCH_SIZE_ROWS}

it fails. Sorry, by original had an extra

-

Also, if I do

batch_size_rows: ${BATCH_SIZE_ROWS}

in the

target-postgres

it works just fine. So I’m thinking it might be something in the target, but I’m not sure how meltano abstracts away the config

https://meltano.com/docs/configuration.html#configuring-settings

export <PLUGIN_NAME>_<SETTING_NAME>=<value>

So something like

export TARGET_REDSHIFT_BATCH_SIZE_ROWS=300000

should do it! The "trick" with Meltano is that you have to include the setting with the tap/target for the setting to be detected in

.env

files, since you're just using environment variables this should just work! Better yet, target-redshift it's one of the included taps/targets in Meltanos discovery which means there's nothing special you need to do, it should just work. I debug this with the command @edgar_ramirez_mondragon gave here https://meltano.slack.com/archives/C01TCRBBJD7/p1633985690450700?thread_ts=1633984552.450600&cid=C01TCRBBJD7

Ok, well I don’t know what I was doing a couple of days ago, but everything that I said wasn’t working is working now. I’m guessing I was doing something silly with target-inheritance or something. Anyways. Thanks a bunch!