Hi, I'm using helm chart to deploy dagster. After I upgrade to 14.20 the dagster daemon cannot start. The error is like this. Any ideas on how to fix this?

2022-06-27 08:32:46 +0000 - dagster.daemon.SchedulerDaemon - ERROR - Caught error, daemon loop will restart:
IsADirectoryError: [Errno 21] Is a directory: '/dagster-workspace/workspace.yaml'

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/daemon.py", line 80, in run_daemon_loop
    result = check.opt_inst(next(daemon_generator), SerializableErrorInfo)
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/daemon.py", line 218, in core_loop
    instance.scheduler.max_tick_retries,
  File "/usr/local/lib/python3.7/site-packages/dagster/scheduler/scheduler.py", line 93, in execute_scheduler_iteration_loop
    log_verbose_checks=(workspace_iteration == 0),
  File "/usr/local/lib/python3.7/site-packages/dagster/scheduler/scheduler.py", line 117, in launch_scheduled_runs
    for location_entry in workspace.get_workspace_snapshot().values()
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 38, in get_workspace_snapshot
    self._location_entries = self._load_workspace()
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 98, in _load_workspace
    origins = self._workspace_load_target.create_origins()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load_target.py", line 27, in create_origins
    return location_origins_from_yaml_paths(self.paths)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load.py", line 35, in location_origins_from_yaml_paths
    workspace_configs = [load_yaml_from_path(yaml_path) for yaml_path in yaml_paths]
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load.py", line 35, in <listcomp>
    workspace_configs = [load_yaml_from_path(yaml_path) for yaml_path in yaml_paths]
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/yaml_utils.py", line 85, in load_yaml_from_path
    with open(path, "r", encoding="utf8") as ff:

2022-06-27 08:32:46 +0000 - dagster.daemon.SchedulerDaemon - ERROR - Caught error, daemon loop will restart:
IsADirectoryError: [Errno 21] Is a directory: '/dagster-workspace/workspace.yaml'

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/daemon.py", line 80, in run_daemon_loop
    result = check.opt_inst(next(daemon_generator), SerializableErrorInfo)
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/daemon.py", line 218, in core_loop
    instance.scheduler.max_tick_retries,
  File "/usr/local/lib/python3.7/site-packages/dagster/scheduler/scheduler.py", line 93, in execute_scheduler_iteration_loop
    log_verbose_checks=(workspace_iteration == 0),
  File "/usr/local/lib/python3.7/site-packages/dagster/scheduler/scheduler.py", line 117, in launch_scheduled_runs
    for location_entry in workspace.get_workspace_snapshot().values()
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 38, in get_workspace_snapshot
    self._location_entries = self._load_workspace()
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 98, in _load_workspace
    origins = self._workspace_load_target.create_origins()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load_target.py", line 27, in create_origins
    return location_origins_from_yaml_paths(self.paths)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load.py", line 35, in location_origins_from_yaml_paths
    workspace_configs = [load_yaml_from_path(yaml_path) for yaml_path in yaml_paths]
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load.py", line 35, in <listcomp>

Hi Tien - when you upgraded, did you also upgrade the helm chart version being used to 0.14.20? The helm chart should be setting a workspace.yaml file for you - but your error message implies that that's not happening. If that's the case, you'll want to be sure that your helm chart version is the same as the dagster version being used by dagit and the daemon

Hi daniel, Helm chart version also updated to 14.20. When I connect to that pod I can see the folder, so not sure why the error

It should be a file not a folder - typically the IsADirectoryError means (confusingly) that it wasn't set as a volume correctly so Docker/K8s tried to be helpful and created it as a folder instead

yes, there is the file

What are its contents?

oh I see,

workspace.yaml

is a folder.

When you say changing the version in the helm chart, can you go into more detail about what you did to upgrade the helm chart?

version: '0.13.16'

to

version: '0.14.20'

And where did you make that change?

in

dagster-helm-release.yaml

, then we deploy it to eks

Got it - the symptoms you're describing would be consistent with it updating the version of dagster being used by the containers in the helm chart, but not actually upgrading the helm chart

so I need to set that in

dagster-helm-release.yaml

?

I'm not actually sure what dagster-helm-release.yaml is referring to exactly

here

That looks correct to me - I'm not familiar with that way of deploying the helm chart though. I wouldn't expect you to need to manually set the configmap or anything like that, the helm chart should be handling it

this is correct right?

That looks right - is that workspace configmap set correctly

I think it's set correctly

So then the question is why is it not available on the daemon pod

I got your point. Will look into that. Thanks for your help

no problem - curious what you find, since this is supposed to be handled automatically for you by the helm chart

yep, will update here if I find sth

I can say pretty confidently that it is working in the general case, so it's most likely something unique about your deployment setup that the helm chart isn't correctly accounting for