Hello, I am trying to convert a bunch of Python tasks in a homespun framework into Dagster. Initially I am using

@asset

to wrap the

None

-returning tasks and using

non_argument_deps=

to specify dependencies. To match what we had before, I have the following requirements regarding execution during runs of a job (feel free to try and talk me out of having these requirements) • It should be possible to tag an asset such that, if any of its dependencies fail, it executes anyway • It should be possible to tag an asset with a time of day (format of this tbd) after which the asset will start to run even if its dependencies have not completed I believe this would require changing

ActiveExecution._update

(here) and somehow getting my altered version where it's needed. Could I get some initial feedback on what I'm looking to do please?

Hey Gil, I wonder if in your use case, it might make sense to just have a scheduled asset job run for C? eg if you have assets A, B, C where C is downstream of A and B you could setup a job to run C no matter what on some regular cron basis

I don't think that works well, generally C should run after A and B are complete and I don't want to have to maintain estimates of runtime for all dependencies!

Right, I guess a slight variation of this would be, you can setup a run for A, B, C (that runs C after A and B - that would the default behavior) But then you could setup a second "sweeper" run of only C. You could even use a custom ScheduleDefinition to have the C-only run occur only if there was not a successful run of A,B,C to avoid un-necessary duplicate runs The way this latter part would work is

@schedule(job=c_only_asset_job, cron_schedule="0 11 * * *")
def sweeper_job_schedule(context: ScheduleEvaluationContext):
    # code to check for A, B, C
    ...context.instance.get_runs(filters=...)

    if not ABC_run:
      return RunRequest(...)

I just would personally hesitate to modify the execution plan, that's pretty far into the internals

Thanks for this, I'll have a play tomorrow.

I think your challenge is that you want the sweeper jobs to run assets that are not stale, right? eg if A, B aren't updated, you still want C to run, even though in that case C is not stale

Good point, they may in fact not be stale at all

Yea I think normally, if C actually depends on A, B, the freshness policies and regular scheduling will get you really far. But if C is only sort of dependent on them, the situation is more complex

Well thanks a lot for the responses.

Yea, as you build out the scenario let us know if theres any feedback or specifics we can get into further.