Hey team! Im playing around with meltano for the first time and trying to set up a basic elt pipeline within a devcontainer on a local environment - I am planning to do this starting with installing ClickHouse. I have meltano as the base image on this devcontainer and i am running

meltano add utility clickhouse

followed by

meltano install

However, the output is as follows

root@0153664ae564:/workspaces/sh-data-platform/src/hub# meltano add utility clickhouse
Added utility 'clickhouse' to your project
Variant:        clickhouse (default)
Repository:     <https://github.com/ClickHouse/ClickHouse/>
Documentation:  <https://hub.meltano.com/utilities/clickhouse--clickhouse>

2025-03-17T19:52:06.382434Z [info     ] Skipped installing utility 'clickhouse'
2025-03-17T19:52:06.382756Z [info     ] Skipped installing 1/1 plugins

To learn more about utility 'clickhouse', visit <https://hub.meltano.com/utilities/clickhouse--clickhouse>
root@0153664ae564:/workspaces/sh-data-platform/src/hub# meltano install
2025-03-17T19:52:11.621647Z [info     ] Installing 1 plugins          
2025-03-17T19:52:11.639125Z [info     ] Skipped installing utility 'clickhouse'
2025-03-17T19:52:11.639525Z [info     ] Skipped installing 1/1 plugins

Any ideas why it keeps skipping the installation of the plugin? Is there a way for me to run these commands with a higher level of verbosity? This is on a fresh container so there shouldnt be any conflict issues.

As you might be able to discern from the logs,

meltano add

will also try to install the plugin automatically (unless

--no-install

is specified), so the proceeding

meltano install

invocation is redundant.

meltano --log-level debug ...

might reveal some more info. If the plugin is installed for you locally and you are copying the

.meltano

directory into the image, it might be picking that up.

Hi @Reuben (Matatika), thank you for that I tried running with the debug level logs and there is still not enough information on why it gets skipped

root@0153664ae564:/workspaces/sh-data-platform/src/hub# meltano --log-level debug  install
2025-03-17T22:49:31.962341Z [debug    ] Meltano 3.6.0, Python 3.9.21, Linux (x86_64)
2025-03-17T22:49:31.968006Z [debug    ] /etc/timezone found, contents:
 Etc/UTC

2025-03-17T22:49:31.968819Z [debug    ] /etc/localtime found          
2025-03-17T22:49:31.970405Z [debug    ] 2 found:
 {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
2025-03-17T22:49:31.989834Z [debug    ] Creating DB engine for project at '/workspaces/sh-data-platform/src/hub' with DB URI 'sqlite:////workspaces/sh-data-platform/src/hub/.meltano/meltano.db'
2025-03-17T22:49:32.238485Z [debug    ] Found plugin parent            parent=clickhouse plugin=clickhouse source=LOCKFILE
2025-03-17T22:49:32.239037Z [info     ] Installing 1 plugins          
2025-03-17T22:49:32.258108Z [info     ] Skipped installing utility 'clickhouse'
2025-03-17T22:49:32.258740Z [info     ] Skipped installing 1/1 plugins

Hmm... I just looked at the code and it seems there isn't any extra info logged out about why plugin installs are skipped (https://github.com/meltano/meltano/blob/main/src/meltano/core/plugin_install_service.py#L536:L542 for reference). Are you finding that you can't run the ClickHouse utility inside the devcontainer?

I tried running that command, output below

root@87e9cfa238c1:/workspaces/sh-data-platform/src/hub# meltano  invoke --containers clickhouse:start
2025-03-18T12:47:56.356438Z [info     ] Environment 'dev' is active   
2025-03-18T12:47:57.154781Z [info     ] Running command in container   container_name=meltano-clickhouse--start-52ab5dd3-2209-40cd-a25b-32fe566eaf3c
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

DockerContainerError(500, 'failed to set up container networking: driver failed programming external connectivity on endpoint meltano-clickhouse--start-52ab5dd3-2209-40cd-a25b-32fe566eaf3c (53250a719f09a380517f39d57148de103ed37fa1d94a5f680063c2d26216de3a): Bind for 0.0.0.0:8123 failed: port is already allocated', 'b38347a03431230b2161e526174b8b9bd52253e7d3a3ddcf12433bfbc6b1008b')

Docker does work within the container, ive tested it. The error is a port conflict, but im not sure how i can change the default port to something else. I tried running

meltano config clickhouse

and this is the output

2025-03-18T12:51:07.023587Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
{}

I don't think they are configurable. You probably have to override

commands

for the utility in your `meltano.yml`:

plugins:
  utilities:
  - name: clickhouse
    variant: clickhouse
    commands:
    start:
      args: start
      container_spec:
        image: clickhouse/clickhouse-server:latest
        ports:
          '8123': '8123'
          '9000': '9000'