Hi guys I m having an issue with schedules I have a repo set

Question

Hi guys I m having an issue with schedules I have a repo setup like this ``` repository def repo return test ScheduleDefinition name= test schedule cron schedule= pipeline name= test ``` And my `dagster yaml` looks like that ```scheduler module dagster cron cron scheduler class SystemCronScheduler``` Looking at crontab and dagit it looks like everything is setup alright yet my test pipeline isn t running at all Any help would be much appreciated

alex · Answer

can you share `dagster schedule debug` output

Alexandre Borrego · Answer

```Traceback most recent call last File opt miniconda3 envs dagster env bin dagster line 8 in lt module gt sys exit main File opt miniconda3 envs dagster env lib python3 8 site packages dagster cli init py line 38 in main cli obj= pylint disable=E1123 File Users alex local lib python3 8 site packages click core py line 829 in call return self main args kwargs File Users alex local lib python3 8 site packages click core py line 782 in main rv = self invoke ctx File Users alex local lib python3 8 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File Users alex local lib python3 8 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File Users alex local lib python3 8 site packages click core py line 1066 in invoke return ctx invoke self callback ctx params File Users alex local lib python3 8 site packages click core py line 610 in invoke return callback args kwargs File opt miniconda3 envs dagster env lib python3 8 site packages dagster cli schedule py line 414 in schedule debug command return execute debug command click echo File opt miniconda3 envs dagster env lib python3 8 site packages dagster cli schedule py line 420 in execute debug command debug info = instance scheduler debug info File opt miniconda3 envs dagster env lib python3 8 site packages dagster core instance init py line 905 in scheduler debug info for schedule state in self all stored schedule state File opt miniconda3 envs dagster env lib python3 8 site packages dagster core instance init py line 962 in all stored schedule state return self schedule storage all stored schedule state repository origin id AttributeError NoneType object has no attribute all stored schedule state ```

alex · Answer

hmmm how about `dagster instance info`

alex · Answer

if cron is running you will find entries in `$DAGSTER HOME schedules logs` you can peak at

Alexandre Borrego · Answer

Oops my bad I gave you the output of a `dagster schedule debug` cmd without setting `$DAGSTER HOME` Here s the actual debug output ```Scheduler Configuration ======================= Scheduler module dagster cron cron scheduler class SystemCronScheduler config Scheduler Info ============== Running Cron Jobs schedules scripts d884eafd2a7e83f4f444f1030ce8e24884deff4b sh gt schedules logs d884eafd2a7e83f4f444f1030ce8e24884deff4b scheduler log 2 gt amp 1 dagster schedule d884eafd2a7e83f4f444f1030ce8e24884deff4b Scheduler Storage Info ====================== simple schedule cron schedule python path opt miniconda3 envs dagster env bin python repository origin id 2e51ed3d6f6bacb42ecd352e253e03fc9a7a2398 repository pointer from dagster poc repo import repo schedule origin id d884eafd2a7e83f4f444f1030ce8e24884deff4b status RUNNING```

Alexandre Borrego · Answer

I have a folder with an id as a name in `$DAGSTER HOME schedules logs` but it s empty

alex · Answer

alright is `dagster poc` installed in to the python environment

Alexandre Borrego · Answer

It is a module that I m loading with ```load from python module dagster poc repo``` in my `workspace yaml` Not sure if that answers the question If I run the test pipeline manually it works btw

alex · Answer

so `cron` will fire from its own working directory does your test work if you do it from a different directory edit im not sure this is a good test since the `dagit` load path is different than the schedule execution

alex · Answer

one solution would be to `pip install e dagster poc` `conda develop`

alex · Answer

or to switch to a file based `load from`

Alexandre Borrego · Answer

Switching to a file based `load from` didn t help I m assuming the crontab will use ` opt miniconda3 envs dagster env bin python` as the python path right

alex · Answer

you ll need to reconcile the schedules after you change the `load from` to get the change to take gt I m assuming the crontab will use opt miniconda3 envs dagster env bin python as the python path right yep but we dont change the working directory so module loads that depend on current working directory will be a problem

Alexandre Borrego · Answer

So I m looking at the generated command trying to understand how this thing works ``` opt miniconda3 envs dagster env bin python m dagster api launch scheduled execution schedule name simple schedule f absolute path dagster dagster poc repo py a repo schedules logs c0ce9a3338520b4e3d934890c94c532aedcd82af $ RUN DATE c0ce9a3338520b4e3d934890c94c532aedcd82af result ``` If I run this myself from the `$DAGSTER HOME` dir it works fine But I get an error if I run it from anywhere else because of the relative path to the log file

Alexandre Borrego · Answer

Ok manually fixing that log file path allowed me to get the error message which I would have expected in dagit ``` class IPCErrorMessage message null serializable error info class SerializableErrorInfo cause null cls name ModuleNotFoundError message ModuleNotFoundError No module named dagster poc n stack ```

alex · Answer

did you set `$DAGSTER HOME` to

Alexandre Borrego · Answer

Should it be set to an absolute path

Alexandre Borrego · Answer

But yes that s how I set it

alex · Answer

ya it should be resolved

alex · Answer

we should catch error on this better since it trips up our system apparently

Alexandre Borrego · Answer

Ok so now I get the error messages in the `schedules logs` dir but they don t show up in dagit Is that expected

alex · Answer

ya i m going to work on fixing it so that we can capture and store the load errors they currently happen right before we enter the scope where errors get written out

Alexandre Borrego · Answer

Alright yeah this would be good Thanks for the quick turn around Is there a piece of documentation you can point me towards when it comes to packaging repos

alex · Answer

unfortunately not yet

Alexandre Borrego · Answer

Ok no problem Really appreciated your help Have a good one

alex · Answer

let me know if you get stuck again and thanks for the feedback definitely a part of the system that needs some improvement