Hey all I am curious if people have sent data to Redshift an Meltano #announcements

Hey all! I am curious if people have sent data to ...

gunnar

02/15/2021, 7:15 PM

Hey all! I am curious if people have sent data to Redshift and if it worked successfully for them? I currently am (using Singer alone), however am switching to use Meltano and new to the tool.

douwe_maan

02/15/2021, 8:50 PM

Hey @gunnar! I know that @pankaj_saini, @ingo_klose, and @robert_stolz have used

target-redshift

recently. Let's see if they have some experiences to share 🙂

gunnar

02/15/2021, 8:55 PM

Amazing! Thank you! My issue seems to be with authorization, and saying my password is incorrect. However, I know it is the correct password, and attempted using other authorized user credentials.

douwe_maan

02/15/2021, 9:06 PM

That's odd 😕

meltano config target-redshift

is showing the credentials correctly?

gunnar

02/15/2021, 9:14 PM

It is! Which is why I am a bit stumped on this (also being new to Meltano hah)

douwe_maan

02/15/2021, 9:15 PM

Have you used this target before without Meltano? Did it work then?

gunnar

02/15/2021, 9:15 PM

Yup! Worked for all of my Singer etl's as I only use the Redshift Target.

gunnar

02/15/2021, 9:16 PM

This is why it is very important for me to figure this out, because I am hoping to use Meltano moving forward.

douwe_maan

02/15/2021, 9:17 PM

All right, that gives us something to go on!

douwe_maan

02/15/2021, 9:18 PM

Can you run

meltano elt

in debug mode as

meltano --log-level=debug elt

, so that we get to see some extra output around how exactly tap and target are being executed?

douwe_maan

02/15/2021, 9:18 PM

Then the log lines around how the target is invoked and its environment may clear some things up.

douwe_maan

02/15/2021, 9:19 PM

@gunnar For good measure, can you share your

target-redshift

definition in

meltano.yml

? Sanitized, of course

gunnar

02/15/2021, 9:19 PM

Sure!

gunnar

02/15/2021, 9:21 PM

Copy code

loaders:
- name: target-redshift
  namespace: target_redshift
  pip_url: target-redshift
  executable: target-redshift
  settings:
  - name: redshift_host
  - name: redshift_port
  - name: redshift_database
  - name: redshift_username
  - name: redshift_password
  - name: redshift_schema
  - name: default_column_length
  - name: target_s3.aws_access_key_id
  - name: target_s3.aws_secret_access_key
  - name: target_s3.bucket
  - name: target_s3.key_prefix
  config:
    default_column_length: 'XXXXXXXX'
    redshift_database: 'XXXXXXXX'
    redshift_host: 'XXXXXXXX'
    redshift_password: 'XXXXXXXX'
    redshift_port: 'XXXXXXXX'
    redshift_schema: 'XXXXXXXX'
    redshift_username: 'XXXXXXXX'
    target_s3.aws_access_key_id: 'XXXXXXXX'
    target_s3.aws_secret_access_key: 'XXXXXXXX'
    target_s3.bucket: 'XXXXXXXX'
    target_s3.key_prefix: 'XXXXXXXX'

douwe_maan

02/15/2021, 9:22 PM

OK, that looks right

gunnar

02/15/2021, 9:26 PM

I originally thought it could be due to some sort of permission/connection error, however, it wouldn't be throwing a password error if that was the case (if I am not mistaken)

douwe_maan

02/15/2021, 9:30 PM

Right. Can you share the exact error and (if available) its stacktrace? And did you have a chance to try https://meltano.slack.com/archives/CFG3C3C66/p1613423882191400?thread_ts=1613416536.189700&cid=CFG3C3C66

douwe_maan

02/15/2021, 9:55 PM

The issue turned out that to be that Meltano interpreted

$r

and

$6

in passwords as environment variable expansion, and would complain that the env vars

and

didn't exist, and strip out those variables from the password 🤦‍♂️

douwe_maan

02/15/2021, 9:58 PM

https://gitlab.com/meltano/meltano/-/issues/2581

ingo_klose

02/16/2021, 7:42 AM

Sorry, only saw that thread now...I had a similar issue with a different tap...one has to be careful with special characters in passwords as the password seems to be interpreted and not sent on as is. I think I solved it by putting the password in the .env file- but I'm not sure anymore

pankaj_saini

02/16/2021, 10:08 AM

I had the similar kind of issue with reading the redshift password from the .env file. But my issue was because I was using dotenv plugin of oh-my-zsh on my terminal(to export .env file in all the projects in my system), which seems to work really odd when you have double quoted env variable value in .env file. ( e.g.

TARGET_REDSHIFT_PASSWORD="<password>"

). Probably dotenv plugin thinks the double quotes are part of env variable value. The dotenv plugin somehow overrides how meltano reads the .env file(and overrides all the env variable values). Disabling the dotenv plugin worked for me.

douwe_maan

02/16/2021, 4:27 PM

I think I solved it by putting the password in the .env file-

@ingo_klose Yeah, that would resolve it, since env var expansion is only applied on values in

meltano.yml

, not the environment.

5 Views

Open in Slack

Previous Next