Hi all, I wonder if you can help.
I am trying to integrate my meltano project as part of the gitlab CI/CD pipeline.
I want to build the docker image for my meltano project every time something changes in the project.
Problem: When building the image from scratch the ci needs to run the meltano install command to install all the dependencies such as extractors/tranformers etc.
This however, takes a considerable amount of time.
How do we tackle this? Should I:
1. build a base image which contains all the plugins. This base docker file should only be triggered if the <meltano_project>plugins/files have changed.
2. Have another Docker file that reads from the base (only triggered if the any files changed).
Are there better approaches or is that the best way forward? Thanks