Hello everyone, I'm trying to run the Meltano default tap-postgres on Airflow in a Docker container. When I run it locally on my M1 MAC it succeeds, but when I deploy the container to a hosted environment, the el job fails on the following error. Does anyone have a suggestion of what the issue could be? Based on stackoverflow issues I've tried the following things already: • checked whether rust & cargo are available on the image (Debian) • added libffi-dev to the packages used in the docker image • made sure the platform of the image is set to linux/amd64 like the hosting provider requires

[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO - ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/usr/local/airflow/include/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/usr/local/airflow/include/meltano/.meltano/run/elt/2023-11-08T221428--tap-postgres--target-postgres/9298fda3-b45d-4237-b0cd-421872c3f8ef/tap.ffe67a7d-9721-40d1-9c3f-f2a9f434370c.config.json', '--discover'] returned 1 with stderr:
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -  Traceback (most recent call last):
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -   File "/usr/local/airflow/include/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 5, in <module>
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -     from tap_postgres.tap import TapPostgres
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -   File "/usr/local/airflow/include/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 12, in <module>
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -     import paramiko
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -   File "/usr/local/airflow/include/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/paramiko/__init__.py", line 22, in <module>
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -     from paramiko.transport import (
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -   File "/usr/local/airflow/include/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/paramiko/transport.py", line 33, in <module>
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -     from cryptography.hazmat.primitives.ciphers import algorithms, Cipher, modes
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -   File "/usr/local/airflow/include/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/cryptography/hazmat/primitives/ciphers/__init__.py", line 11, in <module>
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -     from cryptography.hazmat.primitives.ciphers.base import (
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -   File "/usr/local/airflow/include/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/cryptography/hazmat/primitives/ciphers/base.py", line 10, in <module>
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -     from cryptography.exceptions import (
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -   File "/usr/local/airflow/include/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/cryptography/exceptions.py", line 9, in <module>
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO -     from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
[2023-11-08, 22:14:38 UTC] {subprocess.py:93} INFO - ImportError: /usr/local/airflow/include/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: cannot open shared object file: No such file or directory

Did

meltano install

run successfully?

Yes, the error occus when I run meltano el tap-postgres target-postgres

Are you using the standard Meltano Docker image? If so, try using an image with an older Python version to see if it's a dependency issue. I had to go back to Python 3.8 before all my taps were happy.

Hi @mark_estey, no I'm using an image provided by Astronomer (managed Airflow) (should probably have mentioned that), but I can try to use an image with an older python version to see if that works.

One reason could be where the docker container is running Meltano. It is like to be related to having the required networking routes in place to reach the Postgres database and/or proxy settings in place. It could also be related to the Postgres database having security groups or IP restrictions around client connectivity.

I made it work with an image that uses Python 3.10 (but rather use 3.11 obviously). I don't know why it failed with 3.9 and 3.11 though, so hopefully I can upgrade some time in the future... Thanks for the suggestions everyone!

Glad it's working! As much as I love using the latest Python language features, there are a lot of large libraries that simply don't like 3.11 (which may also vary by platform) so I almost always reach for 3.10 the moment something breaks. 😅