https://linen.dev logo
Join Slack
Powered by
Hello all. Hoping for some assistance here with u...
# random
j
Hello all. Hoping for some assistance here with using Meltano/Dagster (https://hub.meltano.com/utilities/dagster/). I have been able to run 
meltano invoke dagster:start
and access the website locally. However, when I try to do it from inside a container I created, I am unable to connect to it.
✅ 1
My current Dockerfile setup
Copy code
# Step 1 - With Debian...
FROM debian:bullseye as build

# ...Grab and run shell script to obtain ODBC Drivers
COPY ./DockerSetup/install-driver.sh /tmp/install-driver.sh
RUN /tmp/install-driver.sh 

# Step 2: Meltano 3.2 image; with latest 3.10 Python
# <https://hub.docker.com/r/meltano/meltano/tags>
FROM meltano/meltano:v3.2-python3.10

WORKDIR /project

# Install Essentials and Curl
RUN apt-get install --reinstall build-essential -y
RUN apt-get install -y libgssapi-krb5-2
RUN apt-get update && apt-get upgrade -y && apt-get install curl -y

# Grabs the ODBC 17 Driver that was generated from Step 1
COPY --from=build /opt/mssql-tools/bin /opt/mssql-tools/bin
COPY --from=build /opt/microsoft /opt/microsoft
COPY ./DockerSetup/odbcinst.ini /etc/odbcinst.ini

# Grabs all the meltano project files 
COPY . .

# Install any additional requirements
RUN pip install -r requirements.txt

# Perform clean install of meltano with settings.
RUN meltano install --clean

# Copy the dagster.yml configuration file to dagster folder under .meltano
# Rename suffix to YAML for Dagster to find it (treats yml/yaml as different suffixes)
COPY ./orchestrate/dagster/dagster.yml /project/.meltano/dagster/dagster.yaml

# For some reason docker-compose.yml doesn't pick up this var ahead of time? 
# Need to include it in docker file
# <https://stackoverflow.com/questions/58578973/docker-compose-not-passing-environment-variables-to-docker-container>
ENV DBT_CLEAN_PROJECT_FILES_ONLY false

# Install DBT-Snowflake dependencies
RUN meltano run dbt-snowflake:deps

# Expose default port used by dagster
EXPOSE 3000
# gRPC server?
EXPOSE 4000

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

ENTRYPOINT [""]
And my docker-compose
Copy code
version: '3.8'

networks: 
  default:
    # Prototyping only?
    external: true
    name: camis-network

services:
  meltano:
    entrypoint: /bin/bash
    stdin_open: true # -i
    tty: true # -t
    build:
      context: . # Current directory
      dockerfile: ./Dockerfile # Build off of the Dockerfile in the current directory as docker-compose.yml
    restart: unless-stopped
    # Looks into the root .env 
    environment:
      - TAP_MSSQL_USER=${TAP_MSSQL_USER}
      - TAP_MSSQL_PASSWORD=${TAP_MSSQL_PASSWORD}
      - TAP_MSSQL_DATABASE=${TAP_MSSQL_DATABASE}
      - TARGET_SNOWFLAKE_ACCOUNT=${TARGET_SNOWFLAKE_ACCOUNT}
      - TARGET_SNOWFLAKE_USER=${TARGET_SNOWFLAKE_USER}
      - TARGET_SNOWFLAKE_PASSWORD=${TARGET_SNOWFLAKE_PASSWORD}
      - TARGET_SNOWFLAKE_ROLE=${TARGET_SNOWFLAKE_ROLE}
      - TARGET_SNOWFLAKE_WAREHOUSE=${TARGET_SNOWFLAKE_WAREHOUSE}
      - TARGET_SNOWFLAKE_DATABASE=${TARGET_SNOWFLAKE_DATABASE}
      - DBT_SNOWFLAKE_SCHEMA=${DBT_SNOWFLAKE_SCHEMA} # DBT Only
      - DBT_TARGET_PATH=${DBT_TARGET_PATH}
      - DBT_CLEAN_PROJECT_FILES_ONLY=${DBT_CLEAN_PROJECT_FILES_ONLY} # Already populated in Dockerfile, due to bug?
      # continue adding env_vars here as required
    ports:
      - 127.0.0.1:3000:3000
For the time being I've been running the dagster:start command afterwards inside the container manually, and checking localhost with curl I see that the page loaded ok
Copy code
root@478811ee7168:/project# curl <http://localhost:3000>
<!doctype html><html lang="en"><head><meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no"/><meta name="theme-color" content="#000000"/><script type="application/json" id="initialization-data">{"pathPrefix": "", "telemetryEnabled": false}</script><script nonce="01092b999b434709a300880c14ed794d">__webpack_nonce__="01092b999b434709a300880c14ed794d"</script><link rel="manifest" href="/manifest.json" crossorigin="use-credentials"/><link rel="icon" type="image/png" href="/favicon.png"/><link rel="icon" type="image/svg+xml" href="/favicon.svg"/><title>Dagit</title><script defer="defer" src="/static/js/main.a2ed00b7.js" nonce="01092b999b434709a300880c14ed794d"></script><link href="/static/css/main.cc1499ae.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>root@478811ee7168:/project#
So I know there's something I'm doing wrong here in my dockerfile/compose but I can't figure out what
Copy code
PS C:\GitHub\camis\elt-pipeline> docker exec -it elt-pipeline-meltano-1 bash
root@de4cdde883ad:/project# meltano invoke dagster:start
2024-01-04T18:38:15.948201Z [info     ] Environment 'dev' is active
2024-01-04 18:38:17 +0000 - dagit - WARNING - The `dagit` CLI command is deprecated and will be removed in dagster 2.0. Please use `dagster-webserver` instead.
2024-01-04 18:38:21 +0000 - dagster.code_server - INFO - Started Dagster code server for file /project/orchestrate/dagster/repository.py in process 20
2024-01-04 18:38:21 +0000 - dagit - INFO - Serving dagster-webserver on <http://127.0.0.1:3000> in process 16
^ the logs when I run the dagster:start
For posterity: It looks like the answer here is to use a custom command like what was shown in this thread, along with my existing docker files: https://meltano.slack.com/archives/C0695LLCGVC/p1698423257139909
Copy code
plugins:
  utilities:
  - name: dagster
    variant: quantile-development
    pip_url: dagster-ext dagster-dbt dagster-azure pendulum<3
    # Run custom commands (meltano invoke dagster:(COMMAND))
    commands:
      dev:
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 0.0.0.0 -d $REPOSITORY_DIR
        executable: dagster_invoker
🙌 2
2 Views
Open in Slack
PreviousNext
Powered by