hello :wave: I have a scheduled job based on an As...
# ask-community
d
hello 👋 I have a scheduled job based on an Asset group selection. I’ve been experiencing lately some runs “stuck” at Started, no step failures. Some steps are missing completely (i.e. can’t find them in preparing, executing, errored or succeeded). This is not the case for every scheduled run. I’m using Dagster v0.15.8, running on GKE and the job uses the k8s_job_executor. Any clues as to why this is happening? My current fix is to terminate the run and materialize remaining assets from the UI, is there a better way (apart from preventing it from happening at all of course 🙂 )?
j
do you have run monitoring enabled? https://dagster.slack.com/archives/C01U954MEER/p1659267448574749 that might help surface some error messaging
d
no but I can give it a go 👍
I came across several bugs in the UI by the way, for instance partitions showing as missing despite corresponding run having successfully finished..
when is the partition presumed to be completed? when all underlying assets have been materialized? or when all steps have run successfully? I’m getting missing partitions in both cases, also without any run worker crashes. Which component would be the prime suspect for this?
j
The first issue you mention is likely a UI bug, do you mind opening a GH issue and adding some screenshots of what you're seeing? A partition would be complete when the asset is successfully materialized, the materialization status of downstream assets shouldn't determine if the partition is complete as far as i know
d
Here it is: https://github.com/dagster-io/dagster/issues/9557 In my specific case there are multiple assets being materialized, this does happen though. I’ll add more findings if I can