https://linen.dev logo
Join Slack
Powered by
When trying to install `tap-redshift` (default mon...
# troubleshooting
h
When trying to install 
tap-redshift
(default monad-inc variant) I get this issue with psycopg2:
I've already installed psycopg2-binary (think it installed as part of the meltano pip install), and tried 
brew install postgres
as well but neither have worked.
v
Can you share your meltano.yml, there's a seperate venv created per tap and target, it's probably that venv that doesn't have the right dependency (or something local, psycopg does need a local dependency installed 😕 )
h
Copy code
version: 1
default_environment: dev
project_id: 775e36a7-4d19-4b89-8ed7-cfec8c58d2a8
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-redshift
    variant: monad-inc
    pip_url: git+<https://github.com/Monad-Inc/tap-redshift.git>
v
https://github.com/Monad-Inc/tap-redshift/blob/main/setup.py#L54 you could add psycopg-binary here since it looks like you own the repo. or you add it to this line
Copy code
pip_url: git+<https://github.com/Monad-Inc/tap-redshift.git>
For me I needed libpq-dev https://meltano.slack.com/archives/CKHP6G5V4/p1660083457596259?thread_ts=1659552634.170039&amp;cid=CKHP6G5V4 hope that helps
m
what version of psycopg2? I haven’t used 
tap-redshift
, but in 
target-redshift
I needed to upgrade to > 2.9.0 https://github.com/transferwise/pipelinewise-target-redshift/pull/228/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7L23
h
So I got it working in the end by simply 
brew link
ing the postgres@10 (didn't try other versions) I installed to get the local dependencies. It was searching for 
pg_config
binary which is provided by the psotgres install.
@Matt Menzenski on an unrelated note, do you know if 
target-redshift
supports IAM based authentication?
m
ha! that’s exactly what i’ll be figuring out today as a matter of fact - see my message over here https://meltano.slack.com/archives/C01PKLU5D1R/p1678467325105679?thread_ts=1678466724.971049&amp;cid=C01PKLU5D1R
I have not tested IAM based authentication to the database. But I can say that S3 access (necessary for COPY command) via an AWS IAM role (as opposed to access key + secret credentials) does not work
so I will probably be adding that to my fork
m
I see that the thread is similar. I can't install 
target-redshift
. I has some issues with psycopg2-binary. With an installation of 
meltano
, I installed 
psycopg2-binary==2.9.5
, but it looks like 
target-redshift
wants 
psycopg2-binary==2.8.5
. Any suggestions what could I do?
Copy code
ld: library not found for -lssl
      clang: error: linker command failed with exit code 1 (use -v to see invocation)
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> psycopg2-binary

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)
m
FWIW, I ended up forking the project because the maintainers seem unresponsive 🤷🏼 my meltano.yml has
Copy code
- name: target-redshift
      variant: transferwise
      pip_url: git+<https://github.com/menzenski/pipelinewise-target-redshift.git@856f6390805e4d64e743901b5f585f01a93df6a1>
and it’s been working fine for me.
I bet my fork will work for you, if that’s the issue you’re having @mindaugas_nizauskas
m
Yes, I actually did that. And installed 
target-redshift
using 
pip install -e .
. It looks like that it helped, thank you.
m
the recommended approach is to use meltano to install it rather than install it directly with 
pip
. Is there a reason you did it that way?
personally, I tend to add the yml lines directly to meltano.yml and then run 
meltano install
to install everything
m
Got it, I am new to meltano, Thanks for pointing this out
h
@Matt Menzenski you mentioned earlier that role-based IAM auth doesn't work for the COPY? Did you overcome this with your fork?
m
IAM auth for the COPY does work.
We’re running Meltano in kubernetes, and we have an AWS IAM role defined using this terraform module with the kubernetes ServiceAccount can assume. This allows our tap-spreadsheets-anywhere Meltano jobs to access S3. However, this role cannot be assumed by AWS Redshift - that was the issue I had run into that made me think IAM auth for the COPY didn’t work. It does work, but this terraform module creates a role that allows 
sts:AssumeRoleWithWebIdentity
, while the Redshift AWS service uses a different action 
sts:AssumeRole
to assume a role. I didn’t want to fork the terraform module so I ended up creating a new IAM role with full S3 access to the buckets we use for Redshift logging and the Redshift temp directory, and I pass that role’s ARN to the target-redshift plugin (the 
aws_redshift_copy_role_arn
setting). It works fine.
Open in Slack
PreviousNext
Powered by