https://dagster.io/ logo
Join Slack
Powered by
c
p
What gets output when you 
crontab -l
?
c
Copy code
0 3 * * * /opt/dagster/dagster_home/schedules/scripts/pipeline_repository.staging_daily_esri.sh # dagster-schedule: pipeline_repository.staging_daily_esri
p
hmm. and when you click through to 
staging_daily_esri
link, you see no tick attempts?
c
nope
p
what’s the underlying OS?
c
ubuntu-small-latest
p
@sashank ^^
c
strangely there is no 
/var/log/syslog
makes me think cron is not set up correctly on the os
p
very curious
does that 
staging_daily_esri.sh
file exist?
i guess i’d also try editing the crontab manually to check cron… something like 
* * * * * date -u >> /opt/dagster/dagster_home/test_cron_log
c
yup it exists
hm
s
Hey Chris, investigating now
Just setup the same environment as you to try and reproduce
m
@sashank this reminds me of my issue from a few weeks ago
which ended up being a permissions issue
(tho on osx not ubuntu)
s
Hey @Chris Roth, was able to reproduce the issue
c
nice! what was the issue?
s
It seems that cron can’t run anything from 
usr/local/bin
, even python
c
interesting
hm
s
So the crontab and script are running as expected, but 
dagster-graphql
fails to run
Trying a few things to fix
❤️ 1
r
could it be timezone difference in docker container?
it's usually UTC in docker containers
s
Hm idts - I was able to reproduce this with a schedule running every minute
r
I don't see any issue with with entryscript.sh
#!/bin/sh
# This block may be omitted if not packaging a repository with cron schedules #################################################################################################### # see: https://unix.stackexchange.com/a/453053 - fixes inflated hard link count touch /etc/crontab /etc/cron./ service cron start export DAGSTER_HOME=/opt/dagster/dagster_home # Add all schedules dagster schedule up # Restart previously running schedules dagster schedule restart --restart-all-running #################################################################################################### DAGSTER_HOME=/opt/dagster/dagster_home dagit -h 0.0.0.0 -p 3000
However my baseimage in dockerfile is 
FROM continuumio/miniconda3:latest
s
@Res Dev and schedules work as expected for you?
r
yes
c
@sashank did you manage to get the schedule to run after you reproduced this?
s
Hey, managed to do some more digging this weekend and I think I know what you’re running into. @John Mav had run into the same thing and we were able to fix his setup
It seems to be a python path issue - when we load your repository we’re probably running into an import error
I’ll be adding better tooling to surface these errors automatically in the next release, but for now if you have access to the machine running the scheduler, you can run 
crontab -e
and edit your schedule like so:
Copy code
* * * * * /dagter_home/schedules/scripts/my_<http://repo.my|repo.my>_pipeline.sh >> debug_file.txt 2>&1  # dagster-schedule: my_<http://repo.my|repo.my>_pipeline
(We’re adding a small output redirect at the end to a file)
In that file, you’ll probably see an import error like this
Copy code
truncated…
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pipelines'
The workaround for now is just to manually add the path to your folder containing your modules in the file you define your repository definition. For example, if you have a 
repos.py
, add this to the top:
Copy code
SCRIPT_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)))
sys.path.append(SCRIPT_PATH)
^ This is 100% not ideal and we’ll have a cleaner error reporting next release, and happy to jump on a call with you to go through this fix
c
i'm looking at this now, finally. it seems like the issue is that 
dagster-graphql
is actually in 
/usr/local/bin/dagster-graphql
(local)
so i can add a quick fix by doing this: 
ln -s /usr/local/bin/dagster-graphql /usr/bin/dagster-graphql
and then i get 
No module named 'repos'
but it can find graphql now
getting this:
Copy code
{
  "data": {
    "startScheduledExecution": {
      "__typename": "PythonError",
      "cause": null,
      "message": "UnboundLocalError: local variable 'execution_plan_index' referenced before assignment\n",
      "stack": [
        "  File \"/usr/local/lib/python3.8/dist-packages/dagster_graphql/implementation/utils.py\", line 14, in _fn\n    return fn(*args, **kwargs)\n",
        "  File \"/usr/local/lib/python3.8/dist-packages/dagster_graphql/implementation/execution/scheduled_execution.py\", line 138, in start_scheduled_execution\n    raise exc\n",
        "  File \"/usr/local/lib/python3.8/dist-packages/dagster_graphql/implementation/execution/scheduled_execution.py\", line 123, in start_scheduled_execution\n    run, result = _execute_schedule(graphene_info, external_pipeline, execution_params, errors)\n",
        "  File \"/usr/local/lib/python3.8/dist-packages/dagster_graphql/implementation/execution/scheduled_execution.py\", line 169, in _execute_schedule\n    execution_plan_snapshot=execution_plan_index.execution_plan_snapshot,\n"
      ]
    }
  }
}
s
This is in the most recent release? Taking a look
c
yup
👍 1
two issues actually - it's not running the cron jobs, which i am debugging atm - i suspect it is trying to run them in the user's home directory and failing to find repos.py when it should be running it in /opt/dagster/app
s
try redirecting the output of the cron command and seeing that error is showing up
Copy code
*/2 * * * * /opt/dagster/dagster_home/schedules/scripts/my_<http://repo.my|repo.my>_pipeline.sh >> ~/log.txt 2>&1 # dagster-schedule: my_<http://repo.my|repo.my>_pipeline
Just adding the 
>> ~/log.txt 2>&1
at the end
c
just tried this to fix the repos.py not found issue 
ENV PYTHONPATH="$PYTHONPATH:/opt/dagster/app"
now i can run 
/opt/dagster/dagster_home/schedules/scripts/pipeline_repository.test_scheduler.sh
from the root home directory, but it still isn't running on its own
will try what you just suggested
s
let me know what you see
i see what’s going on
whenever you have invalid config for the scheduler, there’a framework error being thrown
which is the one you’re seeing: 
UnboundLocalError: local variable 'execution_plan_index' referenced before assignment
what I would recommend to unblock is make sure your config is valid, and you’ll skip past this error. we’ll have a fix out for this soon
c
interesting
would that be 
scheduler.yaml
?
s
Sorry, I meant the environment dict returned from your schedule definition
c
oh ok
s
There’s a validation error happening with the environment dict you return
c
side question - is scheduler.yaml deprecated since the scheduler config was moved into the 
schedule_defs
s
Yup it’s currently backward compatible, but you should move the schedule defs to the repository definition
Under 
schedule_defs
c
ok great. i deleted it today since i added schedule_defs last week
s
Did you have a separate repository.yaml and scheduler.yaml?
c
i did yeah
my scheduler.yaml was:
Copy code
repository:
  file: repos.py
  fn: pipeline_repository
scheduler:
  file: repos.py
  fn: define_schedules
s
And your repository.yaml was just the repository part?
c
schedule_defs is now:
repository.yaml:
Copy code
repository:
  module: repos
  fn: define_repo
schedule_defs:
Copy code
schedule_defs=[
            ScheduleDefinition(
                name='staging_daily_esri',
                mode='staging',
                cron_schedule='0 3 * * *',
                pipeline_name='esri_pipeline_all',
                environment_dict={},
            ),
            ScheduleDefinition(
                name='test_scheduler',
                mode='staging',
                cron_schedule='* * * * *',
                pipeline_name='reseed_geoserver_pipeline',
                environment_dict={},
            ),
        ],
s
Yup that’s perfect
c
i can't think of what would be wrong with the environment though that causes that cron error
s
Try pasting into the playground?
c
OH, i see what you're saying
i think i know what's going on
s
there’s this open in playground button that will put the config in the playground for you
c
so it nows runs successfully if i run 
/opt/dagster/dagster_home/schedules/scripts/pipeline_repository.test_scheduler.sh
from the home directory - you were right that there was an environment error
but for some reason it's not running on it's own through cron
(i have it set to run every minute)
now debugging - gonna tell cron to log errors to a file
👍 1
s
that should tell you what’s going on
c
nope 😕
i did 
* * * * * /opt/dagster/dagster_home/schedules/scripts/pipeline_repository.test_scheduler.sh > ~/wtf.log # dagster-schedule: pipeline_repository.test_scheduler
wtf.log was created but is empty
s
try piping stderr as well
c
oh good point
s
Copy code
>> ~/log.txt 2>&1
c
that did it, thank you!!
i had to give cron access to environment variables
s
sorry for the rough debugging process - trying to improve this currently
here’s how to pass through the env vars. alternatively, you can source your environment within your crontab to copy all your env vars
c
interesting. i did this: https://stackoverflow.com/a/41938139 which also worked
it seems like the disable button might not be working
s
the turn off schedule button?
c
yup
i tried disabling but it's still going
s
hm i can’t repro that one - try running 
dagster schedule debug
and see if it shows you anything
c
good idea
5 Views
Open in Slack
PreviousNext
Powered by