https://linen.dev logo
Join Slack
Powered by
Has anyone had trouble running meltano in docker o...
# announcements
r
Has anyone had trouble running meltano in docker on a windows machine while mounting your project directory to the Meltano container? Every time I do it, it acts like there are no plugins installed. But if I run without the volume mount, this doesn't happen. Just trying to setup a local development environment that makes it easy to propagate changes!
t
Hmm interesting. I have zero experience running this on Windows so I’m flying blind here. What directory are you in when you’re in the container?
c
@ricky_renner, my guess is that this is a permissions issue. That or there's a file that isn't mount to the correct location. For instance, the working-dir is mounted to 
/project
, but that's not the path on your local. What's the specific error you're getting?
r
@christopher_hooper I'm not getting an error, it is just that when I go to run something, it says that certain plugins aren't installed. I'm wondering if it's something with the 
.meltano/
directory being copied back and forth across the Windows machine and the running container. Especially since I have that in my 
.dockerignore
file. Maybe the solution is just rebuild the image whenever I make changes, but I was just hoping to be able to make changes and immediately run a pipeline in the UI and it have the most up to date 
meltano.yml
file.
I have a 
meltano install
in my Dockerfile after copying over the 
meltano.yml
but still doesn't seem to be working 🤷
t
@douwe_maan @aaronsteers do y’all have any idea?
d
@ricky_renner Are you using the 
meltano/meltano
Docker image and mounting your project directory, or are you building a dedicated Docker image for your project?
r
I'm using that image as the base image
d
Can you show me the exact command you're running that's telling you plugins aren't installed?
With 
docker run
and everything 🙂
a
@ricky_renner - Maybe you’ve already tackled this, but just to be sure - do you know if Docker has been granted access to the C drive? At least in prior versions, I know that Docker did not default to having access on the local drive, and you had to explicitly share it. And more confusing, at least when I was testing several months back, docker didn’t always complain about it. It might mount the volume but then there’s nothing inside it. If you are able to confirm the volume mounted correctly (meaning you can see files in the mounted folder and modifications are reflected both in the host and container), then we can safely cross this off the things to check for.
r
Thanks for the reply @aaronsteers! Yes all the files do get transferred over, and when I make changes to the meltano.yml on my Windows machine, they are reflected in the running container. And thank you for thinking of actually sharing the drive - yes I did do that
I might toy around with this some more... but maybe a quick ten minute screenshare next week would make the most sense?
d
Can you show me the exact command you're running that's telling you plugins aren't installed? With 
docker run
 and everything 🙂
A quick screenshare later today/whenever is also an option
r
Hey @douwe_maan sorry for the response to this. But any chance you or your team is available this afternoon to do a quick screenshare for local setup on a Windows machine?
d
I have very little Windows experience while @aaronsteers has tons 🙂 Can you take this AJ?
a
Hi, @ricky_renner - I’m happy to assist.
r
Awesome thank you @aaronsteers. I basically want to make sure that local development is as streamlined/easy as possible with running Meltano in Windows. Currently, I'm running Meltano in Docker, but if I want to make changes such as add Airflow as an orchestrator or install a new extractor/load, running the 
meltano add ...
command has to propogate those changes back to the 
meltano.yml
file on my Windows machine. How would you recommend doing this? I'm currently trying to mount my local directory to the container, but it's running into issues because it's acting like certain Meltano extractors/loaders aren't installed.
I'm happy to do a screenshare if that'd be easier @aaronsteers, but I also thought it would benefit the community if I posted this above^
a
Thanks, @ricky_renner. Can you paste for me the “docker run” or “docker compose” command you are using, and I’ll take a quick look prior to us jumping on a call?
r
Copy code
version: "3.8"
services:
  bw-meltano-ui:
    build: .
    command: ui
    depends_on: 
      - bw-meltano-db
      - bw-meltano-target-db
    env_file: 
      - ./config/meltano-ui.env.dev
    # environment:
      # - MELTANO_PROJECT_READONLY: 1
    ports:
      - target: 5000
        published: 5000
    networks:
      - bw-meltano-net
    volumes:
      - ./meltano.yml:/project/meltano.yml
      - ./extract/:/project/extract/
      - ./load/:/project/load/
      - ./model/:/project/model/
      - ./orchestrate/:/project/orchestrate/
      - ./transform/:/project/transform/
  bw-meltano-target-db:
    image: "postgres"
    env_file: 
      - ./config/meltano-target-db.env.dev
    networks:
      bw-meltano-net:
    ports:
      - target: 5432
        published: 5432
  bw-meltano-db:
    image: "postgres"
    env_file: 
      - ./config/meltano-db.env.dev
    networks:
      bw-meltano-net:
    ports:
      - target: 5432
        published: 5433

networks:
  bw-meltano-net:
Here's our 
docker-compose.yml
file. Basically, we spin up two databases (one for Meltano, and one for the 
target-postgres
loader). I originally was binding my entire project directory to the 
/project
folder on the container, but I thought maybe that that was screwing up everything in the 
.meltano/
directory since Meltano was letting me know that certain taps/targets weren't installed
a
I see. Thanks for posting that. I’m still reviewing on my end, but on first glance, I notice that the 
volumes:
section is mapped to individual artifact locations, which could be part of the reason other components may be missed.
r
Copy code
ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt . 
RUN pip install -r requirements.txt

# Install all plugins into the `.meltano` directory
COPY ./meltano.yml . 
RUN meltano install

# Pin `discovery.yml` manifest by copying cached version to project root
RUN cp -n .meltano/cache/discovery.yml . 2>/dev/null || :

# Copy over remaining project files
COPY . .

# Expose default port used by `meltano ui`
EXPOSE 5000

ENTRYPOINT ["meltano"]
CMD ["ui"]
Here is our Dockerfile as well
a
I’ll review a few more things on my side and I should be ready to meet in 35 minutes. Would that be 2:30pm your time? I could meet at 2:30 or 3pm your time, if that works for you.
r
2:30 PM EST sounds great!
a
@ricky_renner and I just met and we think we might have gotten it working with a tweak to ordering of steps in the 
Dockerfile
. I also proposed we plan to circle back on Windows use cases during next week’s office hours session. @ricky_renner, do keep us posted if this resolves your issue. I’ll create an issue to research that potential Dockerfile issue in the meanwhile. While we’ve got it fresh, do you mind pasting back the updated Dockerfile text here in this thread?
r
Copy code
ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE

WORKDIR /project
COPY . .

# Install any additional requirements
RUN pip install -r requirements.txt

# Install all plugins into the `.meltano` directory
RUN meltano install

# Expose default port used by `meltano ui`
EXPOSE 5000

ENTRYPOINT ["meltano"]
CMD ["ui"]
Here it is! Thank you for taking the time @aaronsteers
a
Happy to help - and I’m grateful for your time to walk me through it. I’ve logged this issue and created a Draft MR with the Dockerfile update. More testing is needed to confirm, but please do post back if you can confirm the encironment does/doesn’t perform as expected after this update.
10 Views
Open in Slack
PreviousNext
Powered by