Hey all! I am having problems with our

target-redshift

(data-mill variant) loader reading the

max_batch_rows

parameter defined in the

meltano.yml

file. I am trying to understand why the pipeline isn't reading this parameter and was just wondering how Meltano interprets configurations from the YAML file (of loaders in particular) when running a pipeline? Any insight would be greatly appreciated!

Hey @connor_flynn - running

meltano config target-redshift list

should give you all the available settings and their associated values from the meltano.yml. Also

meltano invoke --dump=config target-redshift

will give you the config.json file that is passed to the target during execution. I would run those and make sure everything looks as expected.

Great, thank you @pat_nadolny! I will try those out now!

If that max_batch_rows is making it to the config.json then I'd go to the target repo and make sure there isnt a bug or anything with how its being used

Looks like it is reading from the command

meltano invoke --dump=config <loader>

I will have to see if this is an issue with the target-redshift

Interesting I dont see it referenced in the code anywhere when doing a search https://github.com/datamill-co/target-redshift/search?q=max_batch_rows

Same thing I am seeing! I'm wondering if that's why it is not read from the meltano.yml configs!