Dockerfile:

# <http://registry.gitlab.com/meltano/meltano:latest|registry.gitlab.com/meltano/meltano:latest> is also available in GitLab Registry
ARG MELTANO_IMAGE=meltano/meltano:v2.20.0-python3.10
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# Copy over Meltano project directory
COPY . .

# overwrite dagster.yaml with contents of dagster_azure.yaml
RUN rm -rf ./orchestrate/dagster/dagster.yaml
COPY ./orchestrate/dagster/dagster_azure.yaml ./orchestrate/dagster/dagster.yaml

RUN meltano install

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

# Expose default port used by `meltano ui`
EXPOSE 5000
# Expose port used for postgres connection
EXPOSE 5432

# Expose port used for postgres connection
EXPOSE 3000

ENTRYPOINT ["meltano"]

My dbt models get copied in the

COPY . .

step, but if that comes before the

RUN meltano install

step, then presumably a change to the models means the following steps have to be rerun?

🤦‍♂️ I think I can see where I'm going wrong now.

Well this is a huge improvement. My mistake was putting my dbt models inside the

orchestrate

directory, as this meant the

meltano install

step needed to be carried out with every change to my sql models. I need the models in that location for dagster to pick them up. So I split up the

COPY . .

command into two steps, copying my models as the 2nd step, and placed the

RUN meltano install

command between the two COPY commands. It also made a huge difference to the timing needed for

docker push

too, as it's only pushing the small changes to the dbt layer, not the

RUN meltano install

layer (which is largely unchanged anyway) Resultant Dockerfile:

# registry.gitlab.com/meltano/meltano:latest is also available in GitLab Registry
ARG MELTANO_IMAGE=meltano/meltano:v2.20.0-python3.10
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

COPY meltano.yml logging.yaml ga4_reports.json ./
ADD meltano-yml meltano-yml
ADD plugins plugins
# Copy over Meltano project directory
# COPY . .

RUN meltano install

# then copy dbt models in orchestrate folder
ADD orchestrate orchestrate

# overwrite dagster.yaml with contents of dagster_azure.yaml
RUN rm -rf ./orchestrate/dagster/dagster.yaml
COPY ./orchestrate/dagster/dagster_azure.yaml ./orchestrate/dagster/dagster.yaml

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

# Expose default port used by `meltano ui`
EXPOSE 5000
# Expose port used for postgres connection
EXPOSE 5432

# Expose port used for postgres connection
EXPOSE 3000

ENTRYPOINT ["meltano"]

@Will Da Silva (Arch) beyond

meltano.yml

and

/plugins

dir is there anything else I would need to copy from the project dir? Up until now it has just been

COPY . .

as per stock Dockerfile so have just copied everything?