https://dagster.io/ logo
Join Slack
Powered by
Hi guys I'm having an issue with schedules. I have...
# announcements
a
Hi guys I'm having an issue with schedules. I have a repo setup like this:
Copy code
@repository
def repo():
    return [
        test,
        ScheduleDefinition(
            name="test_schedule",
            cron_schedule="* * * * *",
            pipeline_name="test",
        )
    ]
And my 
dagster.yaml
looks like that:
Copy code
scheduler:
  module: dagster_cron.cron_scheduler
  class: SystemCronScheduler
Looking at crontab and dagit, it looks like everything is setup alright, yet my test pipeline isn't running at all... Any help would be much appreciated!
a
can you share 
dagster schedule debug
output ?
a
Copy code
Traceback (most recent call last):
  File "/opt/miniconda3/envs/dagster_env/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/opt/miniconda3/envs/dagster_env/lib/python3.8/site-packages/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/Users/alex/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/alex/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/alex/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/alex/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/alex/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/alex/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/miniconda3/envs/dagster_env/lib/python3.8/site-packages/dagster/cli/schedule.py", line 414, in schedule_debug_command
    return execute_debug_command(click.echo)
  File "/opt/miniconda3/envs/dagster_env/lib/python3.8/site-packages/dagster/cli/schedule.py", line 420, in execute_debug_command
    debug_info = instance.scheduler_debug_info()
  File "/opt/miniconda3/envs/dagster_env/lib/python3.8/site-packages/dagster/core/instance/__init__.py", line 905, in scheduler_debug_info
    for schedule_state in self.all_stored_schedule_state():
  File "/opt/miniconda3/envs/dagster_env/lib/python3.8/site-packages/dagster/core/instance/__init__.py", line 962, in all_stored_schedule_state
    return self._schedule_storage.all_stored_schedule_state(repository_origin_id)
AttributeError: 'NoneType' object has no attribute 'all_stored_schedule_state'
a
hmmm - how about 
dagster instance info
if cron is running you will find entries in 
$DAGSTER_HOME/schedules/logs
you can peak at
a
Oops, my bad, I gave you the output of a 
dagster schedule debug
cmd without setting 
$DAGSTER_HOME
... Here's the actual debug output:
Copy code
Scheduler Configuration
=======================
Scheduler:
     module: dagster_cron.cron_scheduler
     class: SystemCronScheduler
     config:
       {}


Scheduler Info
==============
Running Cron Jobs:
* * * * * ./schedules/scripts/d884eafd2a7e83f4f444f1030ce8e24884deff4b.sh > ./schedules/logs/d884eafd2a7e83f4f444f1030ce8e24884deff4b/scheduler.log 2>&1 # dagster-schedule: d884eafd2a7e83f4f444f1030ce8e24884deff4b


Scheduler Storage Info
======================
simple_schedule:
  cron_schedule: '* * * * *'
  python_path: /opt/miniconda3/envs/dagster_env/bin/python
  repository_origin_id: 2e51ed3d6f6bacb42ecd352e253e03fc9a7a2398
  repository_pointer: from dagster_poc.repo import repo
  schedule_origin_id: d884eafd2a7e83f4f444f1030ce8e24884deff4b
  status: RUNNING
I have a folder with an id as a name in 
$DAGSTER_HOME/schedules/logs
but it's empty
a
alright, is 
dagster_poc
installed in to the python environment ?
a
It is a module that I'm loading with:
Copy code
load_from:
  - python_module: dagster_poc.repo
in my 
workspace.yaml
. Not sure if that answers the question. If I run the test pipeline manually it works btw.
a
so 
cron
will fire from its own working directory, does your test work if you do it from a different directory? edit: im not sure this is a good test since the 
dagit
load path is different than the schedule execution
one solution would be to 
pip install -e dagster_poc
/ 
conda develop
or to switch to a file based 
load_from
a
Switching to a file based 
load_from
didn't help. I'm assuming the crontab will use 
/opt/miniconda3/envs/dagster_env/bin/python
as the python path right?
a
you’ll need to reconcile the schedules after you change the 
load_from
to get the change to take
I’m assuming the crontab will use /opt/miniconda3/envs/dagster_env/bin/python as the python path right?
yep, but we dont change the working directory, so module loads that depend on current working directory will be a problem
a
So I'm looking at the generated command trying to understand how this thing works:
Copy code
/opt/miniconda3/envs/dagster_env/bin/python -m dagster api launch_scheduled_execution --schedule_name simple_schedule -f /absolute/path/dagster/dagster_poc/repo.py -a repo "./schedules/logs/c0ce9a3338520b4e3d934890c94c532aedcd82af/${RUN_DATE}_c0ce9a3338520b4e3d934890c94c532aedcd82af.result"
If I run this myself from the 
$DAGSTER_HOME
dir, it works fine. But I get an error if I run it from anywhere else because of the relative path to the log file...
Ok manually fixing that log file path allowed me to get the error message which I would have expected in dagit:
Copy code
{"__class__": "IPCErrorMessage", "message": null, "serializable_error_info": {"__class__": "SerializableErrorInfo", "cause": null, "cls_name": "ModuleNotFoundError", "message": "ModuleNotFoundError: No module named 'dagster_poc'\n", "stack": ...
a
did you set 
$DAGSTER_HOME
to ‘.’ ?
a
Should it be set to an absolute path?
But yes, that's how I set it.
a
ya it should be resolved
we should catch / error on this better since it trips up our system apparently
a
Ok so now I get the error messages in the 
schedules/logs
dir, but they don't show up in dagit. Is that expected?
a
ya i’m going to work on fixing it so that we can capture and store the load errors - they currently happen right before we enter the scope where errors get written out
a
Alright yeah, this would be good. Thanks for the quick turn around! Is there a piece of documentation you can point me towards when it comes to packaging repos?
a
unfortunately not yet
a
Ok, no problem. Really appreciated your help! Have a good one!
a
let me know if you get stuck again, and thanks for the feedback - definitely a part of the system that needs some improvement
👍 1
2 Views
Open in Slack
PreviousNext
Powered by