Hi all, I'm trying to install the

target-chromadb

loader plugin on an Amazon Linux AMI

Amazon Linux 2023 AMI 2023.4.20240513.0 x86_64 HVM kernel-6.1

It a clean AMI with only python

Python 3.9.16

and Meltano installed

meltano, version 3.4.2

but when I try to add the loader:

meltano add loader target-chromadb

I get the following error:

#meltano add loader target-chromadb
Added loader 'target-chromadb' to your project
Variant:        meltanolabs (default)
Repository:     <https://github.com/meltanolabs/target-chromadb>
Documentation:  <https://hub.meltano.com/loaders/target-chromadb--meltanolabs>

2024-05-26T12:03:38.369567Z [info     ] Installing loader 'target-chromadb'
2024-05-26T12:04:16.909225Z [info     ] Logged pip install output to /root/meltano-projects/my-meltano-project/.meltano/logs/pip/loaders/target-chromadb/pip.log
2024-05-26T12:04:16.910040Z [error    ] Loader 'target-chromadb' could not be installed: Failed to install plugin 'target-chromadb'.
2024-05-26T12:04:16.910307Z [info     ] Collecting git+<https://github.com/meltanolabs/target-chromadb.git>
  Cloning <https://github.com/meltanolabs/target-chromadb.git> to /tmp/pip-req-build-p44vx0cs
  Running command git clone --filter=blob:none --quiet <https://github.com/meltanolabs/target-chromadb.git> /tmp/pip-req-build-p44vx0cs
  Resolved <https://github.com/meltanolabs/target-chromadb.git> to commit 712daa1161055faf2f2bfa4c451655200045866c
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting chromadb<0.4.0,>=0.3.21 (from meltanolabs-target-chromadb==0.0.1)
  Downloading chromadb-0.3.29-py3-none-any.whl.metadata (6.9 kB)
.
.
.
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 150.6/150.6 kB 7.7 MB/s eta 0:00:00
Collecting hnswlib>=0.7 (from chromadb<0.4.0,>=0.3.21->meltanolabs-target-chromadb==0.0.1)
  Downloading hnswlib-0.8.0.tar.gz (36 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting clickhouse-connect>=0.5.7 (from chromadb<0.4.0,>=0.3.21->meltanolabs-target-chromadb==0.0.1)
  Downloading clickhouse_connect-0.7.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.9 kB)
.
.
.
Downloading exceptiongroup-1.2.1-py3-none-any.whl (16 kB)
Downloading sniffio-1.3.1-py3-none-any.whl (10 kB)
Building wheels for collected packages: meltanolabs-target-chromadb, hnswlib, memoization
  Building wheel for meltanolabs-target-chromadb (pyproject.toml): started
  Building wheel for meltanolabs-target-chromadb (pyproject.toml): finished with status 'done'
  Created wheel for meltanolabs-target-chromadb: filename=meltanolabs_target_chromadb-0.0.1-py3-none-any.whl size=5454 sha256=9e0e0eedc94d61a6acb49d3bd80e9647f3591c8b7cb3c19d85be67643cd47ade
  Stored in directory: /tmp/pip-ephem-wheel-cache-t0uvdnn_/wheels/69/e4/24/f0892dc743c88c97ff1868e3f59f95a171f8e4bdbb620c51c3
  Building wheel for hnswlib (pyproject.toml): started
  error: subprocess-exited-with-error
  
  × Building wheel for hnswlib (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for hnswlib (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for hnswlib
  Building wheel for memoization (setup.py): started
  Building wheel for memoization (setup.py): finished with status 'done'
  Created wheel for memoization: filename=memoization-0.4.0-py3-none-any.whl size=50451 sha256=a0aeb8897bea7305b9ecb3c855bfbd790793006ee1569daa5343285e90efa7ff
  Stored in directory: /root/.cache/pip/wheels/a7/cb/f2/25802a0599b033b24cbf2cb0210af6f13c27291db92cd5178c
Successfully built meltanolabs-target-chromadb memoization
Failed to build hnswlib
ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)
#

I've taken out elements of the output to ensure it fits in a single post. I've also found some mentioned of using

export HNSWLIB_NO_NATIVE=1

, but still get the same error? Any one got any ideas?

have you done a check for the dependencies to build

Thanks, I've not done that. Let me give that a try.

I think this is a known issue with the hnswlib dependency: https://github.com/MeltanoLabs/target-chromadb/issues/11