Challenge #2 - Working with Taps that require more than pip for installation Some taps, such as tap-mssql, are not pip installable. In the tap-mssql case, it's a java/clojure app and actually not python based. With these, I have had success in the past, either calling the tap through

docker run

and piping the output downstream as per usual, or else building a custom

tap-to-target

composite docker image on top of the custom image. (Practically speaking, this requires that only the tap or target is custom, and not both.) However we tackle this, I can't personally think of any elegant solution that doesn't use docker - and whatever the solution, we would need to be able to register the required config setting in

discovery.yml

. Option 2A: Similar to "pip_url" for python installs, we add a discovery.yml option for

docker_image

(pointing to a publicly available docker image), and/or

dockerfile_path

(pointing to a dockerfile within in this repo or to another repo). Option 2B: _____ (Any other solutions I haven't thought of yet...) 🤔

I think a

docker_image

option makes sense, along with different installation and invocation code paths for Docker-based plugins as opposed to Pip-based ones. Meltano would require Docker to be running to use these, and the dockerd socket path would need to be configurable. When containerizing a Meltano project, we'd be running a Docker container from inside a Docker container, so we may need to look into "Docker in Docker" (dind): https://hub.docker.com/_/docker

Cool. Thanks for this link. I'll take a look. I'm not really sure yet of the ramifications of dind - but it sounds like we are aligned on support for

docker_image

as a config method for taps/targets in discovery.yml. What if we take tap-mssql as a case study and create an issue/PR to work on bringing that tap into the Meltano platform/ecosystem?

Sounds great, please do! 😄

Logged: https://gitlab.com/meltano/meltano/-/issues/2205