https://linen.dev logo
Join Slack
Powered by
Hello everyone I’m working on meltano upgrading v...
# troubleshooting
v
Hello everyone I’m working on meltano upgrading v1.105.0 -> v2.2. In my ELT both source and target are Postgres DB and as extractor and loader I use transferwise variants. All configuration live in .env file. After upgrade I updated config section to explicitly set extractor and loader configs (Screens with before and after configurations are in thread). When I run I notice weird behaviour that loader saved result in source database. When I tried to change loader env variables to v2 recommended style (TARGET_POSTGRES_*) I received different error 
meltano.core.settings_store.ConflictingSettingValueException: Conflicting values for setting found in: ['TARGET_POSTGRES_HOST', 'POSTGRES_HOST']
When I checked configuration from meltano side I found that both extractor and loader have same config alias (Screen in thread) Finally I tried change loader to datamil-co variant and everything worked but much slower So question, is it something wrong in my configuration or is it some conflict between transferwise extractor and loader. Thanks in advance
message has been deleted
q
The environment variables and the config block in yaml do the same thing. The thing you're doing where you pass in an env var as a config block is probably causing confusion here.
Env variables take precedence over config settings. I'd set everything except password in config, and then set passwords by env variable. You can do it however you like, but I would definitely avoid passing env variables as config arguments, with slight variations of each other. https://docs.meltano.com/guide/configuration#configuration-layers
I'd also remove every env variable that starts with PG_... In the migration guide you can see that those all get replaced with TARGET_POSTGRES_... or TAP_POSTGRES_... its much clearer that way. So for your target-postgres drop that whole 
settings:
block and either set those values just by the default env variable or by config, rather than using custom env variables and then aliasing them
Hope I'm explaining this well lol. But yeah basically: 1. Remove the settings block under target-postgres 2. Remove all the places where you set a config value with an env variable (like 
dbname: $PG_DATABASE
) 3. Rename all of your target-postgres environment variables from "PG_..." to "TARGET_POSTGRES_..." 4. Keep your tap-postgres environment variables as they are, since they're compliant with the expected values (TAP_POSTGRES_...) Now all of your settings will be correctly picked up from the env variables
v
thanks for your response! Unfortunately it did not work for me as all configs which I set as environement variable for target-postgress are not discovered. And when I tried to set explicitly config with CLI I receive conflict again
message has been deleted
q
You haven't removed the old POSTGRES_* variables from your environment. That's what the conflict is.
v
That is the issue, I don’t set this variable
q
if you run echo $POSTGRES_DBNAME, you don't see anything?
v
yes
q
Well... I can confirm that that's very weird. Sorry I cant be more helpful than that lol
And you removed the whole "settings" block in here? The string POSTGRES_DBNAME doesnt exist anywheere in your meltano.yml?
Super weird. Sorry cant help more w troubleshooting but good luck 🙏
v
thank you for your help
Open in Slack
PreviousNext
Powered by