Hello, I've got a bit of an interesting problem that I haven't had much luck in solving. We use the

tap-braze

plugin which relies on Airbyte. Recently (as of this PR it seems), airbyte stopped making their pypi package available (based on their registry), forcing meltano to use the docker image. I currently use our existing Airflow to orchestrate meltano jobs. A

KubernetesPodOperator

using the following arguments

arguments=["run", "tap-braze", "target-postgres"],

runs the image that we built which contains our project. Up until about a week ago, when said PR was merged, this was working just fine (I assume because it was using the pypi package with no problem). Now the

tap-braze

plugin fails with launching the airbyte docker image since we're running our Airflow cluster on EKS. The Airbyte wrapper assumes usage of

docker

even though

OCI_RUNTIME

is being overridden. I'm running on EKS 1.31 which no longer uses docker as a runtime but rather

containerd

. I've worked around this by installing

nerdctl

into my project docker image (as it should be a drop-in replacement for docker cli) and `ln`'ing

nerdctl

to

docker

. I've also gone ahead and mounted the following host directories to get

nerdctl

to at least be able to pull the airbyte image. • /tmp • /run/containerd • /var/lib/containerd • /var/lib/nerdctl My problem is that I still get the following error message and I'm kinda lost lol, my next guess is that the host running the airbyte image needs to have

nerdctl

installed but I'm hoping someone else has come across something similar and solved this in a different way.

time="2025-02-06T000356Z" level=fatal msg="failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: fork/exec /usr/bin/nerdctl: no such file or directory: unknown"

Slack Conversation