https://meltano.com/ logo
#announcements
Title
# announcements
p

proud-pillow-55935

03/11/2021, 2:30 PM
Has anyone had trouble running meltano in docker on a windows machine while mounting your project directory to the Meltano container? Every time I do it, it acts like there are no plugins installed. But if I run without the volume mount, this doesn't happen. Just trying to setup a local development environment that makes it easy to propagate changes!
s

salmon-salesclerk-77709

03/11/2021, 3:07 PM
Hmm interesting. I have zero experience running this on Windows so I’m flying blind here. What directory are you in when you’re in the container?
m

microscopic-dream-75195

03/11/2021, 5:41 PM
@proud-pillow-55935, my guess is that this is a permissions issue. That or there's a file that isn't mount to the correct location. For instance, the working-dir is mounted to
/project
, but that's not the path on your local. What's the specific error you're getting?
👍 1
p

proud-pillow-55935

03/11/2021, 8:29 PM
@microscopic-dream-75195 I'm not getting an error, it is just that when I go to run something, it says that certain plugins aren't installed. I'm wondering if it's something with the
.meltano/
directory being copied back and forth across the Windows machine and the running container. Especially since I have that in my
.dockerignore
file. Maybe the solution is just rebuild the image whenever I make changes, but I was just hoping to be able to make changes and immediately run a pipeline in the UI and it have the most up to date
meltano.yml
file.
I have a
meltano install
in my Dockerfile after copying over the
meltano.yml
but still doesn't seem to be working 🤷
s

salmon-salesclerk-77709

03/11/2021, 9:03 PM
@ripe-musician-59933 @salmon-actor-23953 do y’all have any idea?
r

ripe-musician-59933

03/11/2021, 9:05 PM
@proud-pillow-55935 Are you using the
meltano/meltano
Docker image and mounting your project directory, or are you building a dedicated Docker image for your project?
p

proud-pillow-55935

03/11/2021, 9:06 PM
I'm using that image as the base image
r

ripe-musician-59933

03/11/2021, 9:07 PM
Can you show me the exact command you're running that's telling you plugins aren't installed?
With
docker run
and everything 🙂
s

salmon-actor-23953

03/11/2021, 9:20 PM
@proud-pillow-55935 - Maybe you’ve already tackled this, but just to be sure - do you know if Docker has been granted access to the C drive? At least in prior versions, I know that Docker did not default to having access on the local drive, and you had to explicitly share it. And more confusing, at least when I was testing several months back, docker didn’t always complain about it. It might mount the volume but then there’s nothing inside it. If you are able to confirm the volume mounted correctly (meaning you can see files in the mounted folder and modifications are reflected both in the host and container), then we can safely cross this off the things to check for.
p

proud-pillow-55935

03/11/2021, 9:22 PM
Thanks for the reply @salmon-actor-23953! Yes all the files do get transferred over, and when I make changes to the meltano.yml on my Windows machine, they are reflected in the running container. And thank you for thinking of actually sharing the drive - yes I did do that
I might toy around with this some more... but maybe a quick ten minute screenshare next week would make the most sense?
r

ripe-musician-59933

03/11/2021, 9:23 PM
Can you show me the exact command you're running that's telling you plugins aren't installed? With 
docker run
 and everything 🙂
A quick screenshare later today/whenever is also an option
p

proud-pillow-55935

03/16/2021, 5:31 PM
Hey @ripe-musician-59933 sorry for the response to this. But any chance you or your team is available this afternoon to do a quick screenshare for local setup on a Windows machine?
r

ripe-musician-59933

03/16/2021, 5:32 PM
I have very little Windows experience while @salmon-actor-23953 has tons 🙂 Can you take this AJ?
👍 1
s

salmon-actor-23953

03/16/2021, 5:40 PM
Hi, @proud-pillow-55935 - I’m happy to assist.
🙌 1
p

proud-pillow-55935

03/16/2021, 5:47 PM
Awesome thank you @salmon-actor-23953. I basically want to make sure that local development is as streamlined/easy as possible with running Meltano in Windows. Currently, I'm running Meltano in Docker, but if I want to make changes such as add Airflow as an orchestrator or install a new extractor/load, running the
meltano add ...
command has to propogate those changes back to the
meltano.yml
file on my Windows machine. How would you recommend doing this? I'm currently trying to mount my local directory to the container, but it's running into issues because it's acting like certain Meltano extractors/loaders aren't installed.
I'm happy to do a screenshare if that'd be easier @salmon-actor-23953, but I also thought it would benefit the community if I posted this above^
s

salmon-actor-23953

03/16/2021, 5:50 PM
Thanks, @proud-pillow-55935. Can you paste for me the “docker run” or “docker compose” command you are using, and I’ll take a quick look prior to us jumping on a call?
p

proud-pillow-55935

03/16/2021, 5:51 PM
Copy code
version: "3.8"
services:
  bw-meltano-ui:
    build: .
    command: ui
    depends_on: 
      - bw-meltano-db
      - bw-meltano-target-db
    env_file: 
      - ./config/meltano-ui.env.dev
    # environment:
      # - MELTANO_PROJECT_READONLY: 1
    ports:
      - target: 5000
        published: 5000
    networks:
      - bw-meltano-net
    volumes:
      - ./meltano.yml:/project/meltano.yml
      - ./extract/:/project/extract/
      - ./load/:/project/load/
      - ./model/:/project/model/
      - ./orchestrate/:/project/orchestrate/
      - ./transform/:/project/transform/
  bw-meltano-target-db:
    image: "postgres"
    env_file: 
      - ./config/meltano-target-db.env.dev
    networks:
      bw-meltano-net:
    ports:
      - target: 5432
        published: 5432
  bw-meltano-db:
    image: "postgres"
    env_file: 
      - ./config/meltano-db.env.dev
    networks:
      bw-meltano-net:
    ports:
      - target: 5432
        published: 5433

networks:
  bw-meltano-net:
Here's our
docker-compose.yml
file. Basically, we spin up two databases (one for Meltano, and one for the
target-postgres
loader). I originally was binding my entire project directory to the
/project
folder on the container, but I thought maybe that that was screwing up everything in the
.meltano/
directory since Meltano was letting me know that certain taps/targets weren't installed
s

salmon-actor-23953

03/16/2021, 5:53 PM
I see. Thanks for posting that. I’m still reviewing on my end, but on first glance, I notice that the
volumes:
section is mapped to individual artifact locations, which could be part of the reason other components may be missed.
p

proud-pillow-55935

03/16/2021, 5:53 PM
Copy code
ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt . 
RUN pip install -r requirements.txt

# Install all plugins into the `.meltano` directory
COPY ./meltano.yml . 
RUN meltano install

# Pin `discovery.yml` manifest by copying cached version to project root
RUN cp -n .meltano/cache/discovery.yml . 2>/dev/null || :

# Copy over remaining project files
COPY . .

# Expose default port used by `meltano ui`
EXPOSE 5000

ENTRYPOINT ["meltano"]
CMD ["ui"]
Here is our Dockerfile as well
👍 1
s

salmon-actor-23953

03/16/2021, 5:57 PM
I’ll review a few more things on my side and I should be ready to meet in 35 minutes. Would that be 2:30pm your time? I could meet at 2:30 or 3pm your time, if that works for you.
👍 1
p

proud-pillow-55935

03/16/2021, 6:00 PM
2:30 PM EST sounds great!
👍 1
s

salmon-actor-23953

03/16/2021, 7:28 PM
@proud-pillow-55935 and I just met and we think we might have gotten it working with a tweak to ordering of steps in the
Dockerfile
. I also proposed we plan to circle back on Windows use cases during next week’s office hours session. @proud-pillow-55935, do keep us posted if this resolves your issue. I’ll create an issue to research that potential Dockerfile issue in the meanwhile. While we’ve got it fresh, do you mind pasting back the updated Dockerfile text here in this thread?
👍 2
p

proud-pillow-55935

03/16/2021, 7:32 PM
Copy code
ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE

WORKDIR /project
COPY . .

# Install any additional requirements
RUN pip install -r requirements.txt

# Install all plugins into the `.meltano` directory
RUN meltano install

# Expose default port used by `meltano ui`
EXPOSE 5000

ENTRYPOINT ["meltano"]
CMD ["ui"]
Here it is! Thank you for taking the time @salmon-actor-23953
s

salmon-actor-23953

03/16/2021, 7:45 PM
Happy to help - and I’m grateful for your time to walk me through it. I’ve logged this issue and created a Draft MR with the Dockerfile update. More testing is needed to confirm, but please do post back if you can confirm the encironment does/doesn’t perform as expected after this update.