Hey all, I've got a question about containerization and whether it would be best for my specific use case. The infrastructure I'm developing involves several API taps, as well as on-prem database that I'm accessing using a tap I developed. This tap requires an ODBC driver to be installed and DSN to be configured in the production environment. Obviously the production machine must also be on the organization's network. Would running the production environment in a docker container makes sense given the ODBC requirement? The current ETL infra is running on an on-prem server. It is feasible that an Ubuntu WSL environment could be set up on this server and act as the production environment; would this be a more straightforward solution? I really would like to incorporate CI/CD and testing into the new infrastructure as well, but I'm not sure if there is a workaround that would allow automatic deployment of changes to the infrastructure source code if it is running on a local server. This is my first project involving containers, CI/CD, and Meltano so any advice, guidance, corrections (in the case that anything the question is wrong), etc is welcome. Thanks guys!

Personally I tend towards “always containerize” but I would certainly try for containerization here. Seems to me like you should be able to bake the ODBC driver into the container image. Then you could assert that your local development experience, the CI/CD process, and the production process are all using the same driver installed in the same way (aka via the containerization)

I agree with what Matt said already. I always aim to containerize. It makes for a more consistent environment across local, CI/testing, and prod.

It is feasible that an Ubuntu WSL environment could be set up on this server and act as the production environment; would this be a more straightforward solution?

I havent had to deal with windows servers personally so I dont know the difficulty level but to me this is more of a reason to use containers. As long as the server can run your container then you're good vs if you have to configure the server before the pipelines will work then it feels more fragile. What happens if you want to move to a different server? To the cloud? Other advantages like being able to easily roll back to a previous image version if you have deployment issues, which is harder if you're directly installing on the server.

I really would like to incorporate CI/CD and testing into the new infrastructure as well, but I'm not sure if there is a workaround that would allow automatic deployment of changes to the infrastructure source code if it is running on a local server.

Can you say more about what you mean by this?

In the event that I don't containerize the project, I just wasn't sure how I would be able to orchestrate the automatic deployment of improvements made to the source code of the project automatically to the local production environment.

I've used Ansible in the past to automate deployments onto servers. You can do this with or without ansible but you can configure your deployment automation in github actions/gitlab ci to push your updated code on a merge to main, thats usually a good workflow

As someone who only installed docker for the first time about six months ago, I found it relatively straightforward to get started with a containerised deployment, the existing meltano docs and dockerfile package are super helpful and give you a great starting point. I've customized it a little since then but basically it's the stock dockerfile, then pushed to Azure Container Registry.