Hi all, I am trying to run meltano via docker on our Gitlab CI/CD pipeline. So far, I can mount my meltano project to the docker instance, but for some curious reason it looks like the meltano.yml is not read at all. By running "docker run -v $(pwd)/dbmigrator:/project -w /project meltano/meltano:v1.99.0-python3.7 install" meltano is not installing any plugins and I have to run "docker run -v $(pwd)/dbmigrator:/project -w /project meltano/meltano:v1.99.0-python3.7 add extractor tap-mysql". And when I am running the elt pipeline meltano is complaining about replication method, despite replication-method set to FULL_TABLE in the meltano.yml

If you could share the gitlab yml, and the docker build file you're using, and the error I think it could be a lot more helpful! Generally to debug docker stuff you want to run that docker container locally, connect with bash and run the same commands yourself one and a time and you'll find the issue

Having a look at the contents

meltano.yml

might also help 😉

The thing is, the job is currently running on a machine via cronjob with the exact same meltano.yml or to be more specfiic with the same project directory. So I would rule out that something is off with the meltano.yml

Gotcha. I see you're mounting the

$(pwd)/dbmigrator

directory so I'd try to confirm

meltano.yml

is present in that directory. The entrypoint and workdir seem fine too: https://hub.docker.com/layers/meltano/meltano/v1.99.0-python3.7/images/sha256-05ce82078eb02329a7c823fbd531bf6b507ed1c48f88491789a28d2cd919ae1b

We are using it right now, I would avoid to change the version now while migrating the job from a cronjob where it runs on a random machine to gitlab ci/cd. I would guess that the reason is, that the job is running for 2 years by now and nobody cared to upgrade it.

Gotcha. Does running

docker run

in sequence preserve the container state, or does it create a new container each time?

@Willi Langlitz I see your new posts. You should really point back to this thread I think. We could figure out how to make

docker run

work with a seperate container every time but it really doesn't make a lot of sense to do that. Instead swap the image you're using for the

trigger_data_export

job to be the meltano image. For the repo itself instead of running

meltano add loader

etc why not just setup the

meltano.yml

file (running those commands locally to setup meltano.yml in the way you want) and then you can just run

meltano install

as the first step your task would clean up to be this and your life will be easier

trigger_data_export:
  image:
    name: meltano/meltano # I'd recommend this version if you really want to pin to a version but based on this conversation I'd recomend just sitting with the latest, this one otherwise meltano/meltano:v3.3-python3.10 
    entrypoint: [""]
  stage: export_data
  only:
    - triggers
    - branches
  interruptible: true
  script:
    - meltano install
    - meltano run tap-mysql target-postgres 
  when: manual