https://dagster.io/ logo
Title
g

Gabriel Vieira

09/29/2021, 8:34 PM
Hi ! I'm using dagster via docker-compose but I'm having some issues. Even though I set up the property
auto_remove:true
at file
dagster.yaml
I still have multiple residual containers after its execution. Could anyone help me ? I'm using dagster version 0.12.1
d

daniel

09/29/2021, 11:53 PM
Hi Gabriel - which containers are staying around? Want to post the output of docker ps?
g

Gabriel Vieira

10/01/2021, 12:49 PM
They are generated when I run a pipeline and they are staying around even when the pipeline runs correctly.
c18e8f7e75cf   97ad55e06a9b                      "dagster api execute…"   2 days ago     Exited (137) 2 days ago                                               heuristic_blackburn
9c9d4d164bd0   985ffed1f3f6                      "dagster api execute…"   2 days ago     Exited (0) 2 days ago                                                 blissful_carson
93328949925b   985ffed1f3f6                      "dagster api execute…"   2 days ago     Exited (0) 2 days ago                                                 youthful_dubinsky
67a7d9bc5432   985ffed1f3f6                      "dagster api execute…"   8 days ago     Exited (0) 8 days ago                                                 crazy_heyrovsky
e4021779367a   e1cce5774b6c                      "dagster api execute…"   8 days ago     Exited (0) 8 days ago                                                 sharp_wilson
26d18fe2eee1   94eec2510a6a                      "dagster api execute…"   8 days ago     Exited (0) 8 days ago                                                 youthful_clarke
3af5a64ac7d7   43f0ac3d9086                      "dagster api execute…"   8 days ago     Exited (0) 8 days ago                                                 fervent_satoshi
19b0ff601660   8aa8dfd1382a                      "dagster api execute…"   8 days ago     Exited (0) 8 days ago                                                 condescending_cohen
58ec7f497e90   879ad88f261d                      "dagster api execute…"   8 days ago     Exited (0) 8 days ago                                                 unruffled_heisenberg
4693697dbb26   c58ad8efaf07                      "dagster api execute…"   8 days ago     Exited (0) 8 days ago                                                 boring_bhabha
934746a62d28   8fcb20b10c23                      "dagster api execute…"   9 days ago     Exited (0) 9 days ago                                                 clever_panini
797931514f84   7904f24ad7a5                      "dagster api execute…"   9 days ago     Exited (0) 9 days ago                                                 ecstatic_bardeen
bb1baa4a319d   c98e887b7eb1                      "dagster api execute…"   9 days ago     Exited (0) 9 days ago                                                 gracious_bhabha
c46e778a9ead   727350caba33                      "dagster api execute…"   9 days ago     Exited (0) 9 days ago                                                 hopeful_ishizaka
947f20b2cb0d   727350caba33                      "dagster api execute…"   9 days ago     Exited (0) 9 days ago                                                 condescending_williamson
702a6e849ce1   6129c22d3ecd                      "dagster api execute…"   9 days ago     Exited (0) 9 days ago                                                 sleepy_lederberg
5e1f4e37442b   6129c22d3ecd                      "dagster api execute…"   9 days ago     Exited (0) 9 days ago                                                 angry_volhard
4e501fdbd36e   71e0fe16d767                      "dagster api execute…"   2 weeks ago    Exited (0) 2 weeks ago                                                zen_thompson
43502775fd04   7d9d14e1c278                      "dagster api execute…"   2 weeks ago    Exited (0) 2 weeks ago                                                thirsty_easley
718bf14946bd   7d9d14e1c278                      "dagster api execute…"   2 weeks ago    Exited (0) 2 weeks ago                                                blissful_bhabha
9ac1db437f62   7d9d14e1c278                      "dagster api execute…"   2 weeks ago    Exited (0) 2 weeks ago                                                zen_elgamal
I hope this docker ps is ok to read : P
d

daniel

10/01/2021, 1:54 PM
Would you mind sharing your dagster.yaml?
g

Gabriel Vieira

10/01/2021, 3:06 PM
Sure ! Here it is :
scheduler:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 1

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - DAGSTER_POSTGRES_USER
      - DAGSTER_POSTGRES_PASSWORD
      - DAGSTER_POSTGRES_DB
      - GOOGLE_APPLICATION_CREDENTIALS
    network: dagster_network
    container_kwargs:
      auto_remove: true
      volumes:
        dagster_home:
          bind: /opt/dagster/dagster_home/
        dagster_large_files:
          bind: /opt/dagster/large_files/
        home_config:
          bind: /opt/dagster/.gcloud_config/

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      hostname: dagster_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      hostname: dagster_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      hostname: dagster_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432
I'm also using it via grcp server defined using the tutorial. The
workspace.yaml
is like this:
load_from:
  # Each entry here corresponds to a container that exposes a gRPC server.
  - grpc_server:
      host: 0.0.0.0
      port: 4000
      location_name: "dagster_pipelines_image"
And I'm running it in the pipelines' docker image with:
# to run other commands against your repository
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repository.py"]
Another thing I would like to mention is that I'm using dagstermil to create solids based on jupyter notebooks. I don't know if it changes anything but i think it's worth to mention. Furthermore, I'm struggling with environment variables. Although it shows up inside the pipelines' container, when I run a pipeline and it creates a new container it doesn't have the same env variables.