I m looking at the output of `meltano add files docker` In r Meltano #announcements

I'm looking at the output of `meltano add files do...

julian_knight

06/26/2020, 12:26 AM

I'm looking at the output of

meltano add files docker

. In regards to running

meltano install

in your docker image - does that only install pip dependencies, or does it also make changes to the meltano database? We've been running

meltano install

in a

docker-entrypoint.sh

file thinking it needed access to the database

edgar_ramirez_mondragon

06/26/2020, 1:11 AM

Yeah,

meltano install

also runs db migrations. Its probably best to do that in the entrypoint script like you say. My Dockerfile looks like this (will change to use the official meltano image):

Copy code

FROM python:3.7-slim

## install dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
      libpq-dev \
      python3-dev \
      gcc \
      git

## set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

RUN python -m pip install --upgrade pip && \
    pip install meltano==1.33.0

WORKDIR /usr/src/app

COPY . /usr/src/app

ENTRYPOINT [ "./docker-entrypoint.sh" ]

and the script:

Copy code

#!/usr/bin/env bash
set -e

meltano install

exec "$@"

julian_knight

06/26/2020, 1:14 AM

Thanks. Ours looks very similar

douwe_maan

06/26/2020, 2:51 PM

@julian_knight @edgar_ramirez_mondragon I would actually strongly recommend running

meltano install

as part of your

Dockerfile

, so that you won't be reinstalling all the pip packages every time you run a container, which can significantly increase startup time and can result in different runs using different versions of packages.

douwe_maan

06/26/2020, 2:52 PM

meltano install

runs database migrations, but so does every

meltano

command to make sure the system database has the right tables and columns.

meltano install

doesn't actually use the system database though, it really just iterates over all of the plugins in your

meltano.yml

file and runs

pip install <pip_url>

to install the packages into

.meltano/<plugin_type>/<plugin_name>/venv

. When the image is later used to start a container and run a command, it'll use those pre-installed packages and everything will be fine. If you provide a different

MELTANO_DATABASE_URI

at that point,

meltano

will detect whether there are any pending migrations and apply them then as appropriate.

9 Views

Open in Slack

Previous Next