https://linen.dev logo
Join Slack
Powered by
Anyone tried running the Metano Dockerfile on <gcl...
# infra-deployment
m
Anyone tried running the Metano Dockerfile on gcloud run deploy and had any luck? I've tried a few variations on the standard Dockerfile and keep getting the same error
ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.
The only thing I do know for certain is that there are constraints that require port 8080 to be used, but have not managed to crack it. Does seem like possibly a peculiarity of the Metano Docker image as I also strugged with Heroku to run the Docker image (can explain what I tried separately if anyone interested). FWIW here is the Dockerfile I tried (+numerous alterations) on Gcloud
Copy code
ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt . 
RUN pip install -r requirements.txt

# Install all plugins into the `.meltano` directory
COPY ./meltano.yml . 
RUN meltano install

# Pin `discovery.yml` manifest by copying cached version to project root
RUN cp -n .meltano/cache/discovery.yml . 2>/dev/null || :

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

# Copy over remaining project files
COPY . .

EXPOSE 8080

CMD meltano ui --bind-port=8080 --bind=0.0.0.0
f
I've had to play around with the health probes on my kubernetes deploy quite a bit. I don't know gcloud, but see if there are options to delay the checks. Meltano may take a while to start up, and gcloud may not be giving it enough time before declaring it failed.
m
@fred_reimer it must be that, I tried deploying it on GKE and had no luck either. Will try that
This is the contract that must be met for the container to run, setting the 
--timeout=
setting didn't help. No idea https://cloud.google.com/run/docs/reference/container-contract
f
We use Terraform, but you can parse what I'm successfully using for the meltano ui pod for k8s:
Copy code
readiness_probes = [{
    exec = []
    http_get = [{
      path   = "/"
      port   = 5000
      scheme = "HTTP"
    }]
    tcp_socket            = []
    failure_threshold     = 3
    initial_delay_seconds = 10
    period_seconds        = 30
    success_threshold     = 1
    timeout_seconds       = 15
  }]

  startup_probes = [{
    exec = []
    http_get = [{
      path   = "/"
      port   = 5000
      scheme = "HTTP"
    }]
    tcp_socket            = []
    failure_threshold     = 30
    initial_delay_seconds = 0
    period_seconds        = 5
    success_threshold     = 1
    timeout_seconds       = 15
  }]

  liveness_probes = [{
    exec = []
    http_get = [{
      path   = "/"
      port   = 5000
      scheme = "HTTP"
    }]
    tcp_socket            = []
    failure_threshold     = 3
    initial_delay_seconds = 10
    period_seconds        = 30
    success_threshold     = 1
    timeout_seconds       = 15
  }]
I'm just using: 
command       = ["meltano", "ui"]
We are using the airflow orchestrator, and the web pod for airflow runs on port 8080. Not that Meltano couldn't either, but we use the default port of 5000.
m
Thanks @fred_reimer, I think in my case it seems to be the docker 
CMD
part clashing with the 
ENTRYPOINT
or something, as often the logs display the help text.
Copy code
Usage: meltano [OPTIONS] COMMAND [ARGS]...

  Get help at <https://www.meltano.com/docs/command-line-interface.html>

Options:
  --log-level [debug|info|warning|error|critical]
  -v, --verbose
  --environment TEXT              Meltano environment name
  --version                       Show the version and exit.
  --help                          Show this message and exit.
Open in Slack
PreviousNext
Powered by