https://dagster.io/ logo
Powered by
#announcements
Title
# announcements
k

Ken

08/26/2020, 4:42 PM
Hello guys, I am trying to create a Cron Scheduler, and meeting Error: 
dagster.core.errors.ScheduleExecutionError: Error occurred during the execution of run_config_fn for schedule <name>
More Info in the description!
Dagster version: 0.9.3 dagster.yaml:
Copy code
...

scheduler:
  module: dagster_cron.cron_scheduler
  class: SystemCronScheduler
My Schedule Definition:
Copy code
from dagster import schedule

@schedule(
    pipeline_name="test_pipeline",
    cron_schedule="*/1 * * * *",
    mode='development',
    tags={
        "mode":  'development',
        "pipeline_type": "testing",
        "run_type": "schedule",
        "creator": "ken",
    }
)
def schedule_test_pipeline_development():
    return {}
Stack Trace:
Copy code
dagster.core.errors.ScheduleExecutionError: Error occurred during the execution of run_config_fn for schedule schedule_test_pipeline_development
  File "/usr/local/lib/python3.6/dist-packages/dagster/grpc/impl.py", line 232, in get_external_schedule_execution
    run_config = schedule_def.get_run_config(schedule_context)
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/errors.py", line 192, in user_code_error_boundary
    error_cls(msg_fn(), user_exception=e, original_exc_info=sys.exc_info(), **kwargs), e
  File "/usr/local/lib/python3.6/dist-packages/future/utils/__init__.py", line 403, in raise_from
    exec(execstr, myglobals, mylocals)
  File "<string>", line 1, in <module>

The above exception was the direct cause of the following exception:

TypeError: schedule_test_pipeline_development() takes 0 positional arguments but 1 was given
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/errors.py", line 181, in user_code_error_boundary
    yield
  File "/usr/local/lib/python3.6/dist-packages/dagster/grpc/impl.py", line 232, in get_external_schedule_execution
    run_config = schedule_def.get_run_config(schedule_context)
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/definitions/schedule.py", line 154, in get_run_config
    return self._run_config_fn(context)
What do you think I am missing ?
a

alex

08/26/2020, 4:43 PM
def schedule_test_pipeline_development():
-> 
def schedule_test_pipeline_development(_):
👍 1
k

Ken

08/26/2020, 4:43 PM
trying now...
a

alex

08/26/2020, 4:43 PM
we pass an argument to the function and yours is currently set up to accept 0, resulting in the error 
schedule_test_pipeline_development() takes 0 positional arguments but 1 was given
the 
@schedule
decorator could have better validation here i believe
k

Ken

08/26/2020, 5:44 PM
Hey Alex, Thank you for the pointer, I have 2 things to let you know: 1. https://docs.dagster.io/overview/scheduling-partitions/schedules#cronstring-based-interval:~:text=config.-,schedule_definition.py • The docs page here did not specify that we need to pass "_" as parameter. Perhaps it needs updating ? • The code referenced in that link is not returning the RunConfig Dictionary. Maybe the docs is out-of-sync with the example at: 
dagster/examples/docs_snippets/docs_snippets/overview/schedules_partitions/schedule_definition.py
2. For the dagster-cron: At file: 
dagster/python_modules/libraries/dagster-cron/dagster_cron/cron_scheduler.py
(line 209) We are setting 
export LANG=en_US.UTF-8
But with just that configuration, my cron scheduler does not run, and the scheduler.log file shows: File: scheduler.log
Copy code
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/dagster/__main__.py", line 3, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 760, in main
    _verify_python3_env()
  File "/usr/local/lib/python3.6/dist-packages/click/_unicodefun.py", line 130, in _verify_python3_env
    " mitigation steps.{}".format(extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult <https://click.palletsprojects.com/python3/> for mitigation steps.

This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables:

    export LC_ALL=C.UTF-8
    export LANG=C.UTF-8

Click discovered that you exported a UTF-8 locale but the locale system could not pick up from it because it does not exist. The exported locale is 'en_US.UTF-8' but it is not supported
After replacing that 209 line with
Copy code
export LC_ALL=C.UTF-8
    export LANG=C.UTF-8
The cron-scheduler work as expected. Do you think we should apply this change into the dagster-cron source code ? ( set value "C.UTF-8" for "LC_ALL" and "LANG" ) in the cron_scheduler.py ? Without this, I can still set environment variable to overwrite. But wonder if this is a patch that people in the future would appreciate.
a

alex

08/26/2020, 5:45 PM
cc @sashank
ill fix the docs issue, but I am not certain about the 
cron_scheduler
change
s

sashank

08/26/2020, 5:50 PM
To be honest I can’t remember the reason for why we included that 
LANG
export. I’ll take a look at it today and we will have a fix out for it in tomorrow’s release
a

alex

08/26/2020, 5:51 PM
thanks for the detailed report !
👍 1
k

Ken

08/26/2020, 5:52 PM
Thank you both of you for the quick and helpful response 😄
s

sashank

08/26/2020, 5:53 PM
Yup thanks for digging into this. My plan is to try and remove that 
LANG
export - if you have a chance it would be helpful to confirm that your scheduler works as expected after removing it completely
k

Ken

08/26/2020, 5:58 PM
let me try it now.
hey Sashank, removing that "export LANG" line will result in this error on my end:
Copy code
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/dagster/__main__.py", line 3, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 760, in main
    _verify_python3_env()
  File "/usr/local/lib/python3.6/dist-packages/click/_unicodefun.py", line 130, in _verify_python3_env
    " mitigation steps.{}".format(extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult <https://click.palletsprojects.com/python3/> for mitigation steps.

This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables:

    export LC_ALL=C.UTF-8
    export LANG=C.UTF-8
( as a result, scheduler doesn't work )
s

sashank

08/27/2020, 8:28 PM
Hey @Ken, which OS are you running on? Or are you running this on a particular image that you can share?
k

Ken

08/28/2020, 12:49 AM
Hey Sashank, The image is docker: ubuntu:18.04 ( my os is 20.04, but I run my dagster within the docker )
Powered by