Hi guys, we have a concern on scheduling: our dags...
# announcements
i
Hi guys, we have a concern on scheduling: our dagster scheduler isn’t running — we have done the ff: • Verified that schedules are in crontab • Job runs properly when we manually execute the command listed in crontab • Cron logs are empty (including
mail
) • Jobs don’t show up in SQLite Dagster version: • 0.9.16 Does anyone else have ideas on how we can debug this problem? Thanks a lot!
s
Hey @Iman Encarnacion, sorry that you’re running into this. Can you check the output of
dagster schedule debug
and
dagster schedule logs {schedule_name}
👀 1
i
Hi @sashank, here’s `dagster schedule debug`:
Copy code
Scheduler Configuration
=======================
Scheduler:
     module: dagster_cron.cron_scheduler
     class: SystemCronScheduler
     config:
       {}
Scheduler Info
==============
Running Cron Jobs:
* * * * * /home/ubuntu/CompanyPOC/schedules/scripts/3ad363590579e40e31a4623bee6a41772d9c83b6.sh > /home/ubuntu/CompanyPOC/schedules/logs/3ad363590579e40e31a4623bee6a41772d9c83b6/scheduler.log 2>&1 # dagster-schedule: 3ad363590579e40e31a4623bee6a41772d9c83b6
* * * * * /home/ubuntu/CompanyPOC/schedules/scripts/a7019a763c68d4ee06a7ba15eb7e6bdbcb95132f.sh > /home/ubuntu/CompanyPOC/schedules/logs/a7019a763c68d4ee06a7ba15eb7e6bdbcb95132f/scheduler.log 2>&1 # dagster-schedule: a7019a763c68d4ee06a7ba15eb7e6bdbcb95132f
@hourly /home/ubuntu/CompanyPOC/schedules/scripts/3627fd943a13fed02aeb346f41bfb6fb7b2af937.sh > /home/ubuntu/CompanyPOC/schedules/logs/3627fd943a13fed02aeb346f41bfb6fb7b2af937/scheduler.log 2>&1 # dagster-schedule: 3627fd943a13fed02aeb346f41bfb6fb7b2af937
Scheduler Storage Info
======================
collect_rapidapi_stock_data_schedule:
  cron_schedule: '* * * * *'
  pipeline_origin_id: 3ad363590579e40e31a4623bee6a41772d9c83b6
  python_path: /home/ubuntu/miniconda3/bin/python
  repository_origin_id: 3b6cd935bc861eae729261b9b5e09aa65ce15535
  repository_pointer: -f /home/ubuntu/CompanyPOC/src/Company/repository.py -a Company_hackathon
    -d /home/ubuntu/CompanyPOC
  schedule_origin_id: 3ad363590579e40e31a4623bee6a41772d9c83b6
  status: RUNNING
sync_s3_schedule:
  cron_schedule: '* * * * *'
  pipeline_origin_id: a7019a763c68d4ee06a7ba15eb7e6bdbcb95132f
  python_path: /home/ubuntu/miniconda3/bin/python
  repository_origin_id: 3b6cd935bc861eae729261b9b5e09aa65ce15535
  repository_pointer: -f /home/ubuntu/CompanyPOC/src/Company/repository.py -a Company_hackathon
    -d /home/ubuntu/CompanyPOC
  schedule_origin_id: a7019a763c68d4ee06a7ba15eb7e6bdbcb95132f
  status: RUNNING
transform_sf_schedule:
  cron_schedule: 0 * * * *
  pipeline_origin_id: 3627fd943a13fed02aeb346f41bfb6fb7b2af937
  python_path: /home/ubuntu/miniconda3/bin/python
  repository_origin_id: 3b6cd935bc861eae729261b9b5e09aa65ce15535
  repository_pointer: -f /home/ubuntu/CompanyPOC/src/Company/repository.py -a Company_hackathon
    -d /home/ubuntu/CompanyPOC
  schedule_origin_id: 3627fd943a13fed02aeb346f41bfb6fb7b2af937
  status: RUNNING
@sashank we tried
dagster schedule logs …
and here’s the output:
Copy code
dagster schedule logs /home/ubuntu/CompanyPOC/schedules/logs/3ad363590579e40e31a4623bee6a41772d9c83b6/scheduler.log
And turns out the log is empty:
Copy code
$> cat /home/ubuntu/CompanyPOC/schedules/logs/3ad363590579e40e31a4623bee6a41772d9c83b6/scheduler.log
$>
s
Hm that leads me to believe that the cron jobs aren’t running at all. A good test might be trying to add your own cron job that runs every minute and simply write to a file:
Copy code
* * * * * echo "test" > test.txt
i
hi @sashank forgot to add here that we already found a workaround for the problem, though in a very hacky way. basically, we just made sure that within the dagster-generated
.sh
files in crontab, we add the following line:
cd $DAGSTER_HOME
basically making sure that the
python -m dagster
command runs in
DAGSTER_HOME
. Is this unexpected behavior?
a
why do you need to run in that directory? is that where your python code is? if you need things run in a specific working directory, you can set that argument in your
workspace.yaml
Copy code
load_from:
  - python_file:
      relative_path: hello_world_repository.py
      working_directory: ./
from the debug output - its using
/home/ubuntu/CompanyPOC