https://dagster.io/ logo
#random
Title
w

won

11/09/2021, 1:42 PM
Hi all. i'v got some trouble with my code. Try to Multi-container Docker deployment Deploying Dagster to Docker with mounting volumes (for user_code) run compose and up 4 containers (
docker_postgresql
,
docker_user_code
,
docker_dagit
,
docker_daemon
) See in UI my jobs. Next go to lachpad and run any of jobs. dagster starting new container with same image as my user_code container. Then i get error:
Copy code
dagster.core.errors.DagsterInvariantViolationError: Could not find pipeline 'ozon_fbs_job'. Found: 'my_job'.
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/impl.py", line 75, in core_execute_run
    recon_pipeline.get_definition()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/reconstructable.py", line 112, in get_definition
    defn = self.repository.get_definition().get_pipeline(self.pipeline_name)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/repository.py", line 878, in get_pipeline
    return self._repository_data.get_pipeline(name)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/repository.py", line 658, in get_pipeline
    return self._pipelines.get_definition(pipeline_name)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/repository.py", line 141, in get_definition
    for found_name in self.get_definition_names()
I haven't got job with
my_job
name 🤯 Where is MY jobs? (
ozon_stocks_job
,
ozon_fbs_job
) 👀 project struct:
Copy code
dagster_docker/
├───user_code/
│   ├───ops/
│   │   └───ozon_fbs.py
│   │   └───ozon_stocks.py
│   ├───resoutces/
│   │   └───db_resource.py
│   └───repo.py
├───dagster.yaml
├───docker-compose.yml
├───Dockerfile_dagster
├───Dockerfile_user_code
├───requirements.txt
├───workspace.yaml
Wher did i miss?
repo.py
Copy code
from dagster import repository, schedule
from ops.ozon_fbs import ozon_fbs_job
from ops.ozon_stocks import ozon_stocks_job


@schedule(cron_schedule="1 10 * * *", job=ozon_stocks_job, execution_timezone="US/Central")
def my_schedule(_context):
    return {}


@schedule(cron_schedule="0 10 * * *", job=ozon_fbs_job, execution_timezone="US/Central")
def ozon_daily_schedule(_context):
    run_config = {}
    return run_config

@repository
def deploy_docker_repository():
    return [ozon_stocks_job, ozon_fbs_job, my_schedule, ozon_daily_schedule]
dagster.yaml
Copy code
scheduler:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - DAGSTER_POSTGRES_USER
      - DAGSTER_POSTGRES_PASSWORD
      - DAGSTER_POSTGRES_DB
    container_kwargs:
      volumes:
        - repo.py:/opt/dagster/app/
    network: docker_network

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      hostname: docker_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      hostname: docker_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      hostname: docker_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432
part of docker-compose with volume
Copy code
docker_user_code:
    build:
      context: .
      dockerfile: ./Dockerfile_user_code
    container_name: docker_user_code
    image: docker_user_code_image
    restart: always
    environment:
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
      DAGSTER_CURRENT_IMAGE: "docker_user_code_image"
    volumes: # Make docker client accessible so we can terminate containers from dagit
      - ./user_code:/opt/dagster/app/
    networks:
      - docker_network
and my docker image of
Dockerfile_user_code
Copy code
FROM python:3.7-slim

RUN pip install \
    dagster \
    dagster-postgres \
    dagster-docker

COPY requirements.txt /requirements.txt
RUN pip install --no-cache-dir --requirement /requirements.txt && rm requirements.txt

# Add repository code
WORKDIR /opt/dagster/app
COPY user_code/repo.py /opt/dagster/app

# Run dagster gRPC server on port 4000
EXPOSE 4000

# CMD allows this to be overridden from run launchers or executors that want
# to run other commands against your repository
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py"]
m

max

11/09/2021, 5:08 PM
@johann @rex
d

daniel

11/09/2021, 5:11 PM
@won my_job is most likely the job from the original deploy_docker example - did you possibly start with that example, maybe your user code image needs to be rebuilt?
w

won

11/09/2021, 7:53 PM
Thanks you all!
docker-compose up -d --no-deps --build docker_user_code
fix my trouble. I have another question. After job is done - container exited but still exist. What should i do to make them automatically remove?
m

max

11/09/2021, 7:56 PM
pass
--rm
when you run
docker-compose
❤️ 1
👍 1
d

daniel

11/09/2021, 10:41 PM
you can put this in your run launcher to automatically remove job pods:
Copy code
run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    container_kwargs:
      auto_remove: true
❤️ 1
👍 1
w

won

11/10/2021, 6:12 AM
@daniel Yes! it's just what i need. Thanks!
3 Views