https://dagster.io/ logo
Join Slack
Powered by
Hello! :wave: I would like to ask about docker-com...
# announcements
r
Hello! 👋 I would like to ask about docker-compose file. Is there any available for dagster? I would like to test solution quickly on my local cluster using KIND
f
I have actually created something like this today for my own usage
It shouldn't be a big problem, you can pretty much copy & paste from the documentation and then write a simple docker-compose file
I'm using this right now:
Copy code
version: "3.7"
services:
  db:
    image: postgres:11.5-alpine
    restart: always
    env_file:
      - dev/env-postgres.env
    environment:
      - PGDATA=/var/lib/postgresql/data/pgdata
    volumes:
      - app-db-data:/var/lib/postgresql/data/pgdata
  dagit:
    build:
      context: .
      dockerfile: dagit.dockerfile
    volumes:
      - dagster-home:/opt/dagster/dagster_home
      - dagster-artifacts:/opt/dagster/local
    env_file:
      - dev/env-postgres.env
    ports:
    - 3000:3000
    depends_on:
      - db
volumes:
  dagster-home:
  dagster-artifacts:
Then, 
dagit.dockerfile
is like this:
Copy code
FROM python:3.7-slim

# Cron is required to use scheduling in Dagster
RUN apt-get update && apt-get install -yqq cron

RUN mkdir -p /opt/dagster/dagster_home /opt/dagster/app

RUN pip install dagit dagster-postgres dagster-cron

# Copy your pipeline code and entrypoint.sh to /opt/dagster/app
COPY dagstrolabe/* scripts/entrypoint.sh /opt/dagster/app/

# Copy dagster instance YAML to $DAGSTER_HOME
COPY dagstrolabe/dagster.yaml /opt/dagster/dagster_home/

WORKDIR /opt/dagster/app

RUN chmod +x entrypoint.sh

EXPOSE 3000

ENTRYPOINT ["/opt/dagster/app/entrypoint.sh"]
In this case 
dagstrolabe
is just a dummy name that I picked up for testing
and inside I keep my files
The 
entrypoint.sh
is from the documentation:
Copy code
#!/bin/sh
export DAGSTER_HOME=/opt/dagster/dagster_home

# This block may be omitted if not packaging a repository with cron schedules:
####################################################################################################
# see: <https://unix.stackexchange.com/a/453053> - fixes inflated hard link count
touch /etc/crontab /etc/cron.*/*

service cron start

# Add all schedules defined by the user
dagster schedule up
####################################################################################################

# Launch Dagit as a service
DAGSTER_HOME=/opt/dagster/dagster_home dagit -h 0.0.0.0 -p 3000
r
@Fran Sanchez thank you for respond! Could you also share config files from dev folder?
f
Sure, these only have some env vars for the connection
Copy code
POSTGRES_SERVER=db
POSTGRES_USER=dagit
POSTGRES_PASSWORD=dev_pass_123
POSTGRES_DB=dagit
These are used by the postgres docker image to initialise the database and also are used by dagit to connect to it.
r
Copy code
dagit_1  | Usage: dagster schedule up [OPTIONS]
dagit_1  | 
dagit_1  | Error: No arguments given and workspace.yaml not found.
dagit_1  | Usage: dagit [OPTIONS]
dagit_1  | 
dagit_1  | Error: No arguments given and workspace.yaml not found.
dagster_dagit_1 exited with code 2
I've got this kind of bug - could you please add always dagster yaml instance? Guess I missing it
f
I use a 
workspace.yaml
which seems that you don't have
You need it.
r
Hmm could you send it also? Maybe it's good time to ask owners of product about official dagster image
f
That file doesn't have anything interesting... That's dependant on your repository, mine looks like this but I don't think it's relevant:
Copy code
load_from:
  - python_file: repo.py
I would suggest you, if you haven't already, to follow the tutorial and rea the changes from 0.7.0 to 0.8.0
Open in Slack
PreviousNext
Powered by