have you read this yet https://docs.dagster.io/latest/deploying/instance/ ?

where are the dask workers running?

for context: the way that we have things set up currently is that the source of truth for the “event stream” is the event storage as per configured on the dagster instance. The worker nodes are expected to have the same instance cofiguration so that they can write the event stream to the same place

I am using

dagster_dask

and pointing to a scheduler I started in the config

So the workers are running in AWS along with the scheduler on a master node connected through dask

I guess based on your explanation I should verify that the workers have S3 access too, which I haven't checked because most of my stuff is just on a shared EBS volume

can you hover over the version number in

dagit

?

When I hover over the version, a big tooltip keeps flashing but not staying long enough for me to read it

how about

cat $DAGSTER_HOME/dagster.yaml

if you go through that first link i sent you

to point at that- you should get everything showing up in dagit

For future reference, I had to get

DAGSTER_HOME

propagated to all

dask-workers

for this to work. Can be configured in

jobqueue.yaml

on the Dask side:

# in ~/.config/dask/jobqueue.yaml
sge:
  job-extra: ['-v DAGSTER_HOME=/shared/dagster']

Here

/shared

is a volume that all workers have access to. Also, you have to make sure to start new jobs if you update your source code, otherwise the

dask-workers

will be running stale pipelines

https://github.com/dagster-io/dagster/blob/master/docs/sections/deploying/dask.rst if you are feeling extra generous