Hey I am looking to integrate meltano into an existing airflow docker container but on both MacOS and Linux I’m getting a permission denied when the command

meltano install

is called in Dockerfile. What are peoples suggestions / best practices to integrate meltano and airflow? I have also tried

meltano add utility airflow

but that seems to be broken after doing some research. It also seems more ideal to add a meltano job as part of an airflow process. I’m on meltano version 3.2.0. Dockerfile:

FROM apache/airflow:2.8.0

ENV PATH="/root/.local/bin:$PATH"

ENV PYTHONUNBUFFERED=1
ENV APP_HOME=/app
COPY . $APP_HOME
WORKDIR $APP_HOME

USER root

RUN apt-get update && \
    apt-get install -y --no-install-recommends vim && \
    apt-get autoremove -yqq --purge && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN chown -R airflow $APP_HOME

USER airflow

RUN pip install -U pip
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r requirements.txt

WORKDIR tap-yfinance
RUN pipx ensurepath
RUN pipx install "meltano==3.1.0"
RUN meltano install

WORKDIR $APP_HOME

EXPOSE 8080

CMD ["airflow", "webserver", "-p", "8080"]

Error:

=> ERROR [10/12] RUN pipx install "meltano==3.1.0"                                   23.1s
------                                                                                      
 > [10/12] RUN pipx install "meltano==3.1.0":                                               
0.173 creating virtual environment...                                                       
0.204 creating shared libraries...                                                          
1.898 upgrading shared libraries...
4.385 installing meltano from spec 'meltano==3.1.0'...
23.06 Fatal error from pip prevented installation. Full pip output in file:
23.06     /home/airflow/.local/state/pipx/log/cmd_2024-01-19_03.09.32_pip_errors.log
23.06 
23.06 pip failed to build packages:
23.06     psutil
23.06     backports.zoneinfo
23.06 
23.06 Some possibly relevant errors from pip install:
23.06     error: subprocess-exited-with-error
23.06     error: command 'gcc' failed: Permission denied
23.06 
23.06 Error installing meltano from spec 'meltano==3.1.0'.
------
Dockerfile:27
--------------------
  25 |     WORKDIR tap-yfinance
  26 |     RUN pipx ensurepath
  27 | >>> RUN pipx install "meltano==3.1.0"
  28 |     RUN meltano install
  29 |     
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c pipx install \"meltano==3.1.0\"" did not complete successfully: exit code: 1