Hi! Does anyone know how to properly fix this problem without deleting existing runs from the dagster db?

Note: You can turn off any of following running schedules, but you cannot turn them back on.

hi iman - you shouldn't need to delete any runs to deal with this. What you need to do is press the reconcile button in the UI. which should remove those out-of-date schedules in the UI (usually because they were set up before your code updated and no longer point to the right place) and give you new correct schedules to turn on. (We're working on making this part of the system clearer)

hi @daniel, we’re not seeing a reconcile button:

What command did you run to start dagit?

@daniel we used systemd to start dagit:

"no changes on the code aside from the schedule"" What changes did you make to the schedule

@daniel we changed the schedule from an invalid schedule:

* 0/6 * * *

to

* * * * *

Could you try running "dagster schedule wipe", restart dagit, and see if it still isn't picking up any schedules?

@daniel yes we also tried

wipe

— to clarify, it’s picking up the schedules, and it runs according to schedule, but the UI is giving this warning

Hmmm, that's very strange

we’re not so sure about that since it’s been a while — but to summarize so far: • UI is giving the unloadable warning • We can start/stop the schedules from CLI, but not from web (because of warning) • The schedule is working correctly • Wipe / Restarts / Deleting all state folders and databases doesn’t remove the UI error

Hmmm, that last bullet is the most surprising part. Would you mind pasting the contents of your dagster.yaml?

@daniel here it is:

scheduler:
  module: dagster_cron.cron_scheduler
  class: SystemCronScheduler

telemetry:
  enabled: false

hmmm I am really struggling to come up with a reason that error would be there after wiping and then restarting dagit. Just to triple-check, you're sure that you did that in that order? wipe and then restart?

@daniel yes, that’s correct — we did wipe multiple times, and start/stop also. In fact, we already deleted the

schedules

folder at some point, which I think also deletes the schedules db entirely

got it. if your schedules folder is deleted, you're sure that's the schedules folder that your dagit is using, and restarting dagit still shows the error, I'll need to check with some other folks on the team

@daniel start/stop is dagit indeed. for dagit systemd, we did the usual systemctl stop and start — does that work?

I haven't used systemctl much myself, but I would certainly think so

@daniel we can do one final test, want to clarify that executing these should work: • dagster schedule wipe • dagit restart

Yeah, if the error is still there after doing those two steps and nothing else, that’s surprising and very useful to know

@daniel no worries! thanks for all the help. if ever the error still persists, how can we help you replicate it? maybe just zip the dagster home and send over?

That would be great, yeah

@daniel will do, thanks!!