Hi, I'm using helm chart to deploy dagster. After ...
# ask-community
t
Hi, I'm using helm chart to deploy dagster. After I upgrade to 14.20 the dagster daemon cannot start. The error is like this. Any ideas on how to fix this?
Copy code
2022-06-27 08:32:46 +0000 - dagster.daemon.SchedulerDaemon - ERROR - Caught error, daemon loop will restart:
IsADirectoryError: [Errno 21] Is a directory: '/dagster-workspace/workspace.yaml'

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/daemon.py", line 80, in run_daemon_loop
    result = check.opt_inst(next(daemon_generator), SerializableErrorInfo)
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/daemon.py", line 218, in core_loop
    instance.scheduler.max_tick_retries,
  File "/usr/local/lib/python3.7/site-packages/dagster/scheduler/scheduler.py", line 93, in execute_scheduler_iteration_loop
    log_verbose_checks=(workspace_iteration == 0),
  File "/usr/local/lib/python3.7/site-packages/dagster/scheduler/scheduler.py", line 117, in launch_scheduled_runs
    for location_entry in workspace.get_workspace_snapshot().values()
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 38, in get_workspace_snapshot
    self._location_entries = self._load_workspace()
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 98, in _load_workspace
    origins = self._workspace_load_target.create_origins()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load_target.py", line 27, in create_origins
    return location_origins_from_yaml_paths(self.paths)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load.py", line 35, in location_origins_from_yaml_paths
    workspace_configs = [load_yaml_from_path(yaml_path) for yaml_path in yaml_paths]
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load.py", line 35, in <listcomp>
    workspace_configs = [load_yaml_from_path(yaml_path) for yaml_path in yaml_paths]
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/yaml_utils.py", line 85, in load_yaml_from_path
    with open(path, "r", encoding="utf8") as ff:

2022-06-27 08:32:46 +0000 - dagster.daemon.SchedulerDaemon - ERROR - Caught error, daemon loop will restart:
IsADirectoryError: [Errno 21] Is a directory: '/dagster-workspace/workspace.yaml'

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/daemon.py", line 80, in run_daemon_loop
    result = check.opt_inst(next(daemon_generator), SerializableErrorInfo)
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/daemon.py", line 218, in core_loop
    instance.scheduler.max_tick_retries,
  File "/usr/local/lib/python3.7/site-packages/dagster/scheduler/scheduler.py", line 93, in execute_scheduler_iteration_loop
    log_verbose_checks=(workspace_iteration == 0),
  File "/usr/local/lib/python3.7/site-packages/dagster/scheduler/scheduler.py", line 117, in launch_scheduled_runs
    for location_entry in workspace.get_workspace_snapshot().values()
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 38, in get_workspace_snapshot
    self._location_entries = self._load_workspace()
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 98, in _load_workspace
    origins = self._workspace_load_target.create_origins()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load_target.py", line 27, in create_origins
    return location_origins_from_yaml_paths(self.paths)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load.py", line 35, in location_origins_from_yaml_paths
    workspace_configs = [load_yaml_from_path(yaml_path) for yaml_path in yaml_paths]
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/load.py", line 35, in <listcomp>
🤖 1
d
Hi Tien - when you upgraded, did you also upgrade the helm chart version being used to 0.14.20? The helm chart should be setting a workspace.yaml file for you - but your error message implies that that's not happening. If that's the case, you'll want to be sure that your helm chart version is the same as the dagster version being used by dagit and the daemon
t
Hi daniel, Helm chart version also updated to 14.20. When I connect to that pod I can see the folder, so not sure why the error
d
It should be a file not a folder - typically the IsADirectoryError means (confusingly) that it wasn't set as a volume correctly so Docker/K8s tried to be helpful and created it as a folder instead
Unless there is a
/dagster-workspace/workspace.yaml
file on that pod?
t
yes, there is the file
d
What are its contents?
(it should be a yaml file with an entry for each of your user code deployments, generated by the helm chart)
t
oh I see,
workspace.yaml
is a folder.
but I don't remember changing anything except the version in helm chart?
d
When you say changing the version in the helm chart, can you go into more detail about what you did to upgrade the helm chart?
t
version: '0.13.16'
to
version: '0.14.20'
d
And where did you make that change?
t
in
dagster-helm-release.yaml
, then we deploy it to eks
d
Got it - the symptoms you're describing would be consistent with it updating the version of dagster being used by the containers in the helm chart, but not actually upgrading the helm chart
there's a configmap that it should be setting that includes a workspace.yaml file as a volume on dagit and on the daemon (before 0.14.0, it was only being set on dagit - now, it is being set on both)
t
so I need to set that in
dagster-helm-release.yaml
?
d
I'm not actually sure what dagster-helm-release.yaml is referring to exactly
I don't see that in our provided helm chart anywhere
t
d
That looks correct to me - I'm not familiar with that way of deploying the helm chart though. I wouldn't expect you to need to manually set the configmap or anything like that, the helm chart should be handling it
is it possible to run 'kubectl describe' on your daemon pod and see if the workspace configmap was correctly mounted on it?
t
this is correct right?
d
That looks right - is that workspace configmap set correctly
t
I think it's set correctly
d
So then the question is why is it not available on the daemon pod
If the value is correct and its being mounted correctly
t
I got your point. Will look into that. Thanks for your help
d
no problem - curious what you find, since this is supposed to be handled automatically for you by the helm chart
t
yep, will update here if I find sth
d
I can say pretty confidently that it is working in the general case, so it's most likely something unique about your deployment setup that the helm chart isn't correctly accounting for