Hi all, I wonder if you can help. I am trying to integrate my meltano project as part of the gitlab CI/CD pipeline. I want to build the docker image for my meltano project every time something changes in the project. Problem: When building the image from scratch the ci needs to run the meltano install command to install all the dependencies such as extractors/tranformers etc. This however, takes a considerable amount of time. How do we tackle this? Should I: 1. build a base image which contains all the plugins. This base docker file should only be triggered if the <meltano_project>plugins/files have changed. 2. Have another Docker file that reads from the base (only triggered if the any files changed). Are there better approaches or is that the best way forward? Thanks

A self hosted runner could speed builds up? Otherwise you can prune meltano install, though it reduces whats available to

meltano run

and its probably unnecessary. How many taps/targets and whatnot out of curiosity. Is the build time really that bad?

# Consider explicitness here to avoid installing stuff that you only run locally, ie:
RUN meltano install loader target-... \
    && meltano install extractors tap-1 tap-2 tap-3 \
    && meltano install transformer dbt-...

Hi @alexander_butler, my project is already at a minimum. It just just dbt-query and that alone takes 15minutes. I can imagine if the pipeline grows it will take longer.

are you building on a self hosted runner?

I am running meltano install on my local, its taking quite long to just do the install. So I assume when building docker image from gitlab it would be as long.