Hey we upgraded to `dagster 0 9 1` and we will update very s

Question

Hey we upgraded to `dagster 0 9 1` and we will update very soon to `0 9 2` But we encounter an issue with the k8s deployment of dagit at startup with the Scheduler We use the `SystemCronScheduler` so far but will migrate to the `K8sCronJob` soon and here is our custom dagit startup command injected in helm ```command bin bash c if Values userDeployments enabled template dagster dagit scheduleUpCommand $ else service cron start usr local bin dagster schedule up location companies repository usr local bin dagster schedule start start all location companies repository dagit h 0 0 0 0 p 80 end ``` And here is the logs from the container on k8s replicaset ```Starting periodic command scheduler cron Errors Resolved Schedule company update pipeline schedule is set to be running but the scheduler is not running the schedule Schedule company init pipeline schedule is set to be running but the scheduler is not running the schedule Traceback most recent call last File usr local bin dagster line 8 in lt module gt sys exit main File usr local lib python3 7 site packages dagster cli init py line 38 in main cli obj= pylint disable=E1123 File usr local lib python3 7 site packages click core py line 829 in call return self main args kwargs File usr local lib python3 7 site packages click core py line 782 in main rv = self invoke ctx File usr local lib python3 7 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File usr local lib python3 7 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File usr local lib python3 7 site packages click core py line 1066 in invoke return ctx invoke self callback ctx params File usr local lib python3 7 site packages click core py line 610 in invoke return callback args kwargs File usr local lib python3 7 site packages dagster cli schedule py line 255 in schedule start command return execute start command schedule name start all kwargs click echo File usr local lib python3 7 site packages dagster cli schedule py line 268 in execute start command instance start schedule and update storage state external schedule File usr local lib python3 7 site packages dagster core instance init py line 983 in start schedule and update storage state return self scheduler start schedule and update storage state self external schedule File usr local lib python3 7 site packages dagster core scheduler scheduler py line 271 in start schedule and update storage state name=external schedule name dagster core scheduler scheduler DagsterSchedulerError You have attempted to start schedule company init pipeline schedule but it is already running Loading repository Serving on in process 107 Telemetry As an open source project we collect usage statistics to inform development priorities For more information read We will not see or store solid definitions pipeline definitions modes resources context or any data that is processed within solids and pipelines To opt out add the following to $DAGSTER HOME dagster yaml creating that file if necessary telemetry enabled false Welcome to Dagster If you have any questions or would like to engage with the Dagster team please join us on Slack ``` The problem here is that it trigger a crashloop of the container and dagit is not really up on k8s If someone has an idea on this I think I will remove our custom commands to schedule up and start but do I have to run these commands manually ``` usr local bin dagster schedule up location companies repository usr local bin dagster schedule start start all location companies repository ``` Thanks in advance for your help if you need any info or debug logs let me know

Titouan · Answer

Added some schedule debug info ``` dagster schedule debug Scheduler Configuration ======================= Scheduler module dagster cron cron scheduler class SystemCronScheduler config Scheduler Info ============== Running Cron Jobs 0 1 opt dagster dagster home schedules scripts ebd2d09d4e4f0b995dfdf904538d81dbadd910b9 sh gt opt dagster dagster home schedules logs ebd2d09d4e4f0b995dfdf904538d81dbadd910b9 scheduler log 2 gt amp 1 dagster schedule ebd2d09d4e4f0b995dfdf904538d81dbadd910b9 0 9 opt dagster dagster home schedules scripts 691fc74250cbff690d83461f61335db90f1ed7f7 sh gt opt dagster dagster home schedules logs 691fc74250cbff690d83461f61335db90f1ed7f7 scheduler log 2 gt amp 1 dagster schedule 691fc74250cbff690d83461f61335db90f1ed7f7 Scheduler Storage Info ====================== company init pipeline schedule cron schedule 0 1 pipeline origin id ebd2d09d4e4f0b995dfdf904538d81dbadd910b9 python path usr local bin python repository origin id 22c214dc9a43303ac1e73076f59f8a054b929929 repository pointer m cnty pipeline repository a companies repository schedule origin id ebd2d09d4e4f0b995dfdf904538d81dbadd910b9 status RUNNING company update pipeline schedule cron schedule 0 9 pipeline origin id 691fc74250cbff690d83461f61335db90f1ed7f7 python path usr local bin python repository origin id 22c214dc9a43303ac1e73076f59f8a054b929929 repository pointer m cnty pipeline repository a companies repository schedule origin id 691fc74250cbff690d83461f61335db90f1ed7f7 status RUNNING```

Titouan · Answer

Seems I m missing something about how I load scheduled pipelines

Titouan · Answer

Here my `workspace yaml` has not changed since dagster 0 8 ```load from python package package name cnty pipeline repository attribute example repository python package package name cnty pipeline repository attribute activities repository python package package name cnty pipeline repository attribute companies repository```

johann · Answer

< sashank>

cat · Answer

hmm so we throw the error that you re seeing when a user tries to start a schedule that is already running so removing `schedule start` command should fix it

cat · Answer

~i think this could be caused by the schedule state being set to running in the db dagit being terminated without the schedule state being updated to stopped in the db and then redeploying dagit~

cat · Answer

~which would then think the schedule is still running and also be unable to start the schedule~

cat · Answer

~i think using local file system for storage should resolve this < sashank> feel free to chime in here~

sashank · Answer

The state that stores whether a schedule should be running or not is persistent across deploys If your schedule was already set to be previously running `dagster schedule up` will make sure to put schedules set to be running back on the `Scheduler` If you look at your debug output you can first see the output of `dagster schedule up` which is putting running schedules back on the `Scheduler` But then you re running `dagster schedule start start all` which is causing an error because the schedules are already started

sashank · Answer

All you need to do is take out the `dagster schedule start start all`

Titouan · Answer

ok hand thanks a lot very clear

Titouan · Answer

Here we go ok hand