https://linen.dev logo
Join Slack
Powered by
Hi all, I have been banging my head against a wall...
# docker
j
Hi all, I have been banging my head against a wall trying to get Docker to start Dagster as an entrypoint via Meltano. I am currently using the https://hub.meltano.com/utilities/dagster/ utility. If I run meltano inside the container to invoke dagster to start it up, it works just fine. However if I try 
CMD
or 
ENTRYPOINT
, nothing seems to happen and Docker composes without ever starting Dagster. Looking for thoughts as to what I could be doing wrong here.
Copy code
plugins:
  utilities:
  - name: dagster
    variant: quantile-development
    pip_url: dagster-ext dbt-core~=1.7.0 dbt-snowflake~=1.7.0 pendulum<3
    commands:
      start_local: # Run command if not in Docker
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 127.0.0.1 -d $REPOSITORY_DIR
        executable: dagster_invoker
      start_docker: # Run command if in Docker
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 0.0.0.0 -d $REPOSITORY_DIR
        executable: dagster_invoker
The end of the Dockerfile looks like this
Copy code
# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1
ENV TARGET_SNOWFLAKE_ACCOUNT ${TARGET_SNOWFLAKE_ACCOUNT}
ENV TARGET_SNOWFLAKE_USER ${TARGET_SNOWFLAKE_USER}
ENV TARGET_SNOWFLAKE_PASSWORD ${TARGET_SNOWFLAKE_PASSWORD}
ENV TARGET_SNOWFLAKE_ROLE ${TARGET_SNOWFLAKE_ROLE}
ENV TARGET_SNOWFLAKE_WAREHOUSE ${TARGET_SNOWFLAKE_WAREHOUSE}
ENV TARGET_SNOWFLAKE_DATABASE ${TARGET_SNOWFLAKE_DATABASE} 
ENV DBT_SNOWFLAKE_SCHEMA ${DBT_SNOWFLAKE_SCHEMA}

# Install DBT-Snowflake dependencies and manifest.json (for usage by Dagster)
RUN meltano invoke dbt-snowflake:parse_manifest

ENTRYPOINT ["meltano", "invoke", "dagster:start_docker"]
I've also tried running this as a shell script instead to no avail
Dagster proceeds to load if I exec the same command after the fact from the within the container
e
Hey @joshua_janicas! What command are you using to start the container?
j
docker compose up
e
Oh ok, so what does the 
command
look like for the dagster service?
j
I think you mean this?
Copy code
plugins:
  utilities:
  - name: dagster
    variant: quantile-development
    pip_url: dagster-ext dbt-core~=1.7.0 dbt-snowflake~=1.7.0 pendulum<3
    commands:
      start_local: # Run command if not in Docker
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 127.0.0.1 -d $REPOSITORY_DIR
        executable: dagster_invoker
      start_docker: # Run command if in Docker
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 0.0.0.0 -d $REPOSITORY_DIR
        executable: dagster_invoker
i use 
start_docker
if you mean CMD, i was trying 
["meltano", "invoke", "dagster:start_docker"]
. I am trying it now without meltano to see if that makes a difference with the meltano imgae
e
Oh I meant if you're using compose, you must have a 
docker-compose.yml
in which you can override CMD with 
command
j
Copy code
version: '3.8'

networks: 
  default:
    # Prototyping only?
    external: true
    name: network

services:
  meltano:
    entrypoint: /bin/bash
    stdin_open: true # -i
    tty: true # -t
    build:
      context: . # Current directory
      args:
        - TARGET_SNOWFLAKE_ACCOUNT=${TARGET_SNOWFLAKE_ACCOUNT}
        - TARGET_SNOWFLAKE_USER=${TARGET_SNOWFLAKE_USER}
        - TARGET_SNOWFLAKE_PASSWORD=${TARGET_SNOWFLAKE_PASSWORD}
        - TARGET_SNOWFLAKE_ROLE=${TARGET_SNOWFLAKE_ROLE}
        - TARGET_SNOWFLAKE_WAREHOUSE=${TARGET_SNOWFLAKE_WAREHOUSE}
        - TARGET_SNOWFLAKE_DATABASE=${TARGET_SNOWFLAKE_DATABASE}
        - DBT_SNOWFLAKE_SCHEMA=${DBT_SNOWFLAKE_SCHEMA}
      dockerfile: ./Dockerfile # Build off of the Dockerfile in the current directory as docker-compose.yml
    restart: unless-stopped
    # Looks into the root .env 
    environment:
      - MELTANO_ELT_BUFFER_SIZE=${MELTANO_ELT_BUFFER_SIZE}
      - TAP_MSSQL_USER=${TAP_MSSQL_USER}
      - TAP_MSSQL_PASSWORD=${TAP_MSSQL_PASSWORD}
      - TAP_MSSQL_DATABASE=${TAP_MSSQL_DATABASE}
      - TARGET_SNOWFLAKE_ACCOUNT=${TARGET_SNOWFLAKE_ACCOUNT}
      - TARGET_SNOWFLAKE_USER=${TARGET_SNOWFLAKE_USER}
      - TARGET_SNOWFLAKE_PASSWORD=${TARGET_SNOWFLAKE_PASSWORD}
      - TARGET_SNOWFLAKE_ROLE=${TARGET_SNOWFLAKE_ROLE}
      - TARGET_SNOWFLAKE_WAREHOUSE=${TARGET_SNOWFLAKE_WAREHOUSE}
      - TARGET_SNOWFLAKE_DATABASE=${TARGET_SNOWFLAKE_DATABASE}
      - DBT_SNOWFLAKE_SCHEMA=${DBT_SNOWFLAKE_SCHEMA} # DBT Only
      - DBT_TARGET_PATH=${DBT_TARGET_PATH}
      - DBT_CLEAN_PROJECT_FILES_ONLY=${DBT_CLEAN_PROJECT_FILES_ONLY} # Already populated in Dockerfile, due to bug?
      # continue adding env_vars here as required
    ports:
      - 127.0.0.1:3000:3000
this is what my docker omcpose looks like atm
e
Oh I think 
entrypoint: /bin/bash
is the culprit 🙂
j
😳
oh boy...ok let me try to get rid of that line
👍 1
ha. that worked.
with 
CMD ["invoke", "dagster:start_docker"]
at the end of the dockerfile
thanks Edgar, as always
🫠 i dont know why i never onticed that entrypoint. I'm still a beginner to docker; i assume that was overriding whatever was in the dockerfile?
e
Of course, np!
j
the 
entrypoint
i mean
e
docker is lots of black magic really 😅
2 Views
Open in Slack
PreviousNext
Powered by