https://linen.dev logo
Join Slack
Powered by
Hi everyone, I'm trying to install the loader 'tar...
# troubleshooting
m
Hi everyone, I'm trying to install the loader 'target-redshift' in my docker setup but I am getting this error that pg_config executable is not found? Any advice?
e
Hi @martin_leung! Can you try using the Python 3.8 image instead of 3.10? I see there are binary wheels for the version of psycopg2 the target requires: https://pypi.org/project/psycopg2-binary/2.8.5/#files
m
hi @edgar_ramirez_mondragon I am still getting the same error using the 3.8 image
e
Hmm, can you remove 
.meltano
if it’s present in your computer? In case it’s being mounted with the volume
m
hi @edgar_ramirez_mondragon I removed it then tried to install but it still did not work
@edgar_ramirez_mondragon can you see if you get the same error when trying to install target-redshift on 3.8 image?
u
Hey @martin_leung. I’m able to install target-redshift in `meltano/meltano:v2.19.0-python3.8`:
Copy code
root@3be04c3cbac4:/projects# python --version
Python 3.8.16
root@3be04c3cbac4:/projects# meltano --version
meltano, version 2.19.0
root@3be04c3cbac4:/projects# meltano install
Installing 1 plugins...
Installing loader 'target-redshift'...
Installed loader 'target-redshift'
root@3be04c3cbac4:/projects# cat meltano.yml 
version: 1
default_environment: dev
project_id: 91c3fa85-4527-4693-8cb8-7392c1ce6614
environments:
- name: dev
- name: staging
- name: prod
plugins:
  loaders:
  - name: target-redshift
    variant: transferwise
    pip_url: pipelinewise-target-redshift
m
Hi @edgar_ramirez_mondragon i tried installing from a clean slate, can let me know if i did anything wrong here please, I am using a Mac:
d
@martin_leung Is 
libpq-dev
installed in your Docker image?
m
hey @douwe_maan I havent installed anything yet, whats the best way to install that?
d
You mentioned you’re using Docker, right? Can you share your Dockerfile? Or the docker commands you’re running?
m
I havent altered the dockerfile yet, the only docker commands I've run are in the screenshot: • init my project • install
u
OK, can you share your meltano.yml please?
m
yea its the second screenshot above with the version:1 default_environment: dev .... ....
d
Ah, my bad for missing that. Looks like you’re doing effectively the same thing @edgar_ramirez_mondragon is, but it’s working for him 😕 Edgar, any more ideas?
e
I suspect it may have something to do with your mac being M1
u
I wouldn’t expect that to affect what happens inside the Linux Docker container, but I agree it’s the most obvious difference
m
Is there anybody on the meltano team with an M1 computer that could try to reproduce the error using the commands i showed in the screenshots?
e
<!subteam^S02BCD9FFEF> 👆 if anyone’s got a little bit of bandwidth to give this a try: 1. `meltano.yml`:
Copy code
version: 1
default_environment: dev
project_id: 91c3fa85-4527-4693-8cb8-7392c1ce6614
send_anonymous_usage_stats: false
environments:
- name: dev
- name: staging
- name: prod
plugins:
  loaders:
  - name: target-redshift
    variant: transferwise
    pip_url: pipelinewise-target-redshift
2. 
docker run -v "$(pwd)/meltano.yml":/projects/meltano.yml --name target-redshift -w /projects meltano/meltano:v2.19.0-python3.8
3. 
docker exec -e PIP_ONLY_BINARY=':all:' target-redshift meltano install
w
@edgar_ramirez_mondragon Running the 3rd command on my M1 Mac results in the following:
Copy code
will ~ $ docker exec -e PIP_ONLY_BINARY=':all:' target-redshift meltano install
Installing 1 plugins...
Installing loader 'target-redshift'...
ERROR: Cannot install pipelinewise-target-redshift==1.3.0, pipelinewise-target-redshift==1.4.0, pipelinewise-target-redshift==1.4.1, pipelinewise-target-redshift==1.5.0 and pipelinewise-target-redshift==1.6.0 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>

Loader 'target-redshift' could not be installed: failed to install plugin 'target-redshift'.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)
I wouldn’t expect that to affect what happens inside the Linux Docker container, but I agree it’s the most obvious difference
@douwe_maan As of Meltano 2.19.0 we now additionally publish Docker images for arm64, which M1 Macs will default to using over the amd64 version. Since we're using binary wheels for this, I expect the problem lies in one or more of the packages not providing arm64 wheels
@martin_leung @edgar_ramirez_mondragon Adding 
--platform linux/amd64
to the 
docker run
command seems to have fixed the issue for me, although the execution speed is noticeably slower since it isn't using native instructions now.
u
Aah, that makes sense, I was expecting the image to still be x86.
m
i was able to load from s3 to redshfit by adding --platform linux/amd64 thanks everyone!
Open in Slack
PreviousNext
Powered by