https://dagster.io/ logo
#announcements
Title
# announcements
p

Paul Wyatt

02/11/2021, 2:55 AM
In 0.9.3-0.10.4 migration I'm also seeing the below error, which is quite vexing as I don't think it indicates where the problem is being generated:
Copy code
Operation name: RunsRootQuery

Message: Exactly 5 or 6 columns has to be specified for iteratorexpression.

Path: ["repositoriesOrError","nodes",1,"schedules",1,"futureTicks"]

Locations: [{"line":183,"column":3}]
I'm hopeful that moving off the system cron scheduler will remediate, but any guidance is nonetheless helpful
p

prha

02/11/2021, 3:17 AM
Hi Paul. Do you mind sharing what your schedule definitions look like?
We use a library to calculate future ticks (so that you can preview what future executions from a schedule what look like), but it looks like it’s not happy with one of the cron strings for one of your schedules
p

Paul Wyatt

02/11/2021, 6:12 PM
@prha Example schedule def:
Copy code
def basic_example_pipeline_schedule(preset: PresetDefinition) -> ScheduleDefinition:
    """Defines the schedule for the pipeline above."""
    return ScheduleDefinition(
        name="basic_example_pipeline_every_minute",
        cron_schedule="*/5 * * * *",
        pipeline_name="basic_example_pipeline",
        mode=preset.mode,
        run_config=preset.run_config,
        environment_vars=get_environment_variables_for_scheduler()
    )
preset
looks like
Copy code
PresetDefinition(
        "gcp_env_aware",
        mode="gcp",
        run_config={
            "execution": {"multiprocess": {"config": {"max_concurrent": 4}}},
            "resources": {
                "warehouse": {"config": {"project": "virta-eng-prod"}},
                "cloud_storage": {
                    "config": {
                        "bucket": os.getenv("DEFAULT_STORAGE_BUCKET"),
                        "project": "virta-eng-prod"
                    },
                },
                "io_manager": {"config": {"gcs_bucket": os.getenv("DEFAULT_STORAGE_BUCKET")}},
                "pagerduty": {"config": {"routing_key": PAGERDUTY_KEY}},
            },
        },
    )
👀 1
p

prha

02/11/2021, 6:21 PM
@Paul Wyatt There’s nothing wrong with the
cron_schedule
argument that I see in your example… I suspect there’s a specific cron_schedule string that is generating the error. We do need to catch problematic cron strings earlier (at definition time?) and surface a better error message.
p

Paul Wyatt

02/11/2021, 6:21 PM
All schedules are the same as ones that were running on 0.9.3 successfully.
Could I have introduced an error in the switchover from intermediate storage to the iomanager?
p

prha

02/11/2021, 6:25 PM
I think it’s more likely that it is a
cron_schedule
issue. With
0.10.0
we started using a cron-parsing library
croniter
to project future schedule ticks (new
0.10.0
feature). The error that’s being thrown here is by that library. Previously, with the cron scheduler, I think we just copied the cron schedule to a system crontab, which might have a more permissive structure (e.g. accepts comments)?
p

Paul Wyatt

02/11/2021, 6:26 PM
Ooooh there. might be a schedule followed by a comment. I'll look at the croniter docs and check our schedules 1 by 1
thanks for the followup
p

prha

02/11/2021, 6:28 PM
It’d be very helpful if you could confirm once you find the problematic schedule. In the meantime, I think
0.10.5
is being released today which will prevent the runs page from erroring out due to this issue (though the future schedule ticks view might still error).
👍 1
Also, tracking the work to improve the enigmatic error message to help with debugging here: https://github.com/dagster-io/dagster/issues/3668
p

Paul Wyatt

02/11/2021, 6:38 PM
@prha Found it. We were using an empty string for a placeholder schedule that isn't intended to run yet. Croniter doesn't like that but it perfectly comfortable with Feb 31st, so I'm rebuilding with that.
p

prha

02/11/2021, 6:40 PM
wow, very interesting. I guess this worked before because crontab allows for invalid lines, and just doesn’t run them?
p

Paul Wyatt

02/11/2021, 6:42 PM
The prior version of dagster cron scheduler simply wasn't writing them to the crontab