Hi, I have just implemented the scheduler in the ...
# announcements
n
Hi, I have just implemented the scheduler in the latest version, no problem with the daily_scheduler however when I run the cron based one it doesn't seem to pass the mode onto the execution. I get the following error
Copy code
2021-01-21 15:41:31 - SchedulerDaemon - ERROR - Failed to fetch execution plan for success_email_schedule: dagster.core.errors.DagsterInvalidConfigError: Error in config for pipeline success_email_pipeline
    Error 1: Missing required config entry "solids" at the root.

Stack Trace:
  File "c:\python\dagster\venv\lib\site-packages\dagster\grpc\impl.py", line 328, in get_external_execution_plan_snapshot
    create_execution_plan(
  File "c:\python\dagster\venv\lib\site-packages\dagster\core\execution\api.py", line 668, in create_execution_plan
    environment_config = EnvironmentConfig.build(pipeline_def, run_config, mode=mode)
  File "c:\python\dagster\venv\lib\site-packages\dagster\core\system_config\objects.py", line 128, in build
    raise DagsterInvalidConfigError(
My code for the daily_schedule (which works) is:
Copy code
@daily_schedule(
    pipeline_name="summary_email_pipeline",
    start_date=datetime.datetime(2021, 1, 21),
    execution_timezone="Europe/London",
    mode="prod",
    execution_time=datetime.time(21, 00),
)
def summary_email_schedule(date):
    return
However the one which doesn't is...
Copy code
@schedule(
    cron_schedule="*/1 * * * *", pipeline_name="success_email_pipeline", execution_timezone="Europe/London",
    mode="prod",
)
def success_email_schedule(context):
    return
Any ideas? Have been trying different things, can't seem to crack it.
d
Hi Nick - it looks like one schedule is using summary_email_pipeline but the other is using success_email_pipeline. Do those pipelines have different configuration requirements? My takeaway from the error message is that success_email_pipeline may require some solid configuration that it's not receiving from the schedule function.
n
Hi, thanks for the reply. When I run success_email_pipeline as @daily_schedule it works as intended. I can also run it using the CLI, and within dagster itself. Just the cron which it doesn't like (or I have messed up the code somehow)
d
got it - what's the return value from the schedule function? Is it returning nothing, or was the config map that's returned from the schedule function omitted?
and one follow-up question. When you run it within dagit that pipeline doesn't require any configuration? It's just when you run it as a schedule that it complains about missing solids config?
n
Within dagit, I also use the preset "prod"
execution:
multiprocess:
config:
max_concurrent: 1
solids:
success_email:
inputs:
email_addresses:
value: <mailto:no@no.com|no@no.com>
subject:
value: Dagster - Pipeline Successes
storage:
filesystem: {}
Just trying to work out what the return value is
d
Hmmm, I don't think that schedules support presets (need to double check). So you may need to specify that config within your schedule function. The thing that's mysterious to me still is why it's working when you use daily_schedule
(i.e. i'd expect the return value from your schedule function to be something like {"solids": {"success_email": {"inputs": {.etc.}}}})
s
n
Hi, that link was really useful. Daniel, you were right my return was malformed. All sorted now. Thanks!
🎉 1
condagster 1
For anyone with the same problem in the future I did something like this...
@daily_schedule(
pipeline_name="daily_checks_pipeline",
start_date=datetime.datetime(2021, 1, 25),
execution_timezone="Europe/London",
mode="prod",
execution_time=datetime.time(7, 30),
should_execute=weekday_filter,
)
def daily_checks_weekday_schedule(date):
return daily_checks_pipeline.get_preset("prod").run_config
❤️ 1