Hey, we upgraded to

dagster 0.9.1

(and we will update very soon to

0.9.2

). But we encounter an issue with the k8s deployment of dagit at startup with the Scheduler. We use the

SystemCronScheduler

(so far but will migrate to the

K8sCronJob

soon) and here is our custom dagit startup command (injected in helm):

command: [
            "/bin/bash",
            "-c",
            {{- if .Values.userDeployments.enabled }}
            "{{ template "dagster.dagit.scheduleUpCommand" $ }}"
            {{- else }}
            "service cron start; \
            /usr/local/bin/dagster schedule up --location 'companies_repository'; \
            /usr/local/bin/dagster schedule start --start-all --location 'companies_repository'; \
            dagit -h 0.0.0.0 -p 80"
            {{- end }}
          ]

And here is the logs from the container on k8s replicaset:

Starting periodic command scheduler: cron.
Errors Resolved:
Schedule company_update_pipeline_schedule is set to be running, but the scheduler is not running the schedule.
Schedule company_init_pipeline_schedule is set to be running, but the scheduler is not running the schedule.
Traceback (most recent call last):
  File "/usr/local/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster/cli/schedule.py", line 255, in schedule_start_command
    return execute_start_command(schedule_name, start_all, kwargs, click.echo)
  File "/usr/local/lib/python3.7/site-packages/dagster/cli/schedule.py", line 268, in execute_start_command
    instance.start_schedule_and_update_storage_state(external_schedule)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 983, in start_schedule_and_update_storage_state
    return self._scheduler.start_schedule_and_update_storage_state(self, external_schedule)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/scheduler/scheduler.py", line 271, in start_schedule_and_update_storage_state
    name=external_schedule.name
dagster.core.scheduler.scheduler.DagsterSchedulerError: You have attempted to start schedule company_init_pipeline_schedule, but it is already running
Loading repository...
Serving on <http://0.0.0.0:80> in process 107

  Telemetry:

  As an open source project, we collect usage statistics to inform development priorities. For more
  information, read <https://docs.dagster.io/install/telemetry>.

  We will not see or store solid definitions, pipeline definitions, modes, resources, context, or
  any data that is processed within solids and pipelines.

  To opt-out, add the following to $DAGSTER_HOME/dagster.yaml, creating that file if necessary:

    telemetry:
      enabled: false


  Welcome to Dagster!

  If you have any questions or would like to engage with the Dagster team, please join us on Slack
  (<https://bit.ly/39dvSsF>).

The problem here is that it trigger a crashloop of the container and dagit is not really up on k8s. If someone has an idea on this. I think I will remove our custom commands to schedule up and start but do I have to run these commands manually?

/usr/local/bin/dagster schedule up --location 'companies_repository'; \
/usr/local/bin/dagster schedule start --start-all --location 'companies_repository'; \

Thanks in advance for your help, if you need any info or debug logs, let me know.

hmm, so we throw the error that you’re seeing when a user tries to start a schedule that is already running so removing

schedule start

command should fix it

The state that stores whether a schedule should be running or not is persistent across deploys. If your schedule was already set to be previously running,

dagster schedule up

will make sure to put schedules set to be running back on the

Scheduler

. If you look at your debug output, you can first see the output of

dagster schedule up

which is putting running schedules back on the

Scheduler

. But then you’re running

dagster schedule start --start-all

which is causing an error, because the schedules are already started.

👌 thanks a lot, very clear!