Hey! Does anyone have experience with the cluster-autoscaler evicting Dagster op pods (

dagster-step

) in order to scale down a node? 🧵

Working with Michel on this - adding some more errors from subsequent runs:

clustered_extract_tumor_mutation_profile_workflow (06dfdcb4-ac74-4846-8894-a4f3157151a2) started a new run worker while the run was already in state DagsterRunStatus.STARTED. This most frequently happens when the run worker unexpectedly stops and is restarted by the cluster. Marking the run as failed.

Noticed weirdness with the dagster-run pod for this job. It definitely took a long time to spin up the pod. Once the pod actually spun up - it caused this error to be printed and the run was killed.

Hi fahad - when we've seen that happen in the past (another run worker being spun up) it's almost always being done by kubernetes itself rather than dagster - it evicts the pod and then incorrectly tries to spin another copy up, so we detect that early and stop it so that it doesn't try to re-run steps. There's config that you can set that will tell dagster to try to safely resume where it left off when this happened though: https://docs.dagster.io/deployment/run-monitoring#resuming-runs-after-run-worker-crashes-experimental

Awesome thank you for linking the feature! I think we have some investigation to do on our cluster - something must be killing the pod running the dagster-run

Mmmm you've said that your Dagster ops use the node local storage. Are you sure the pods aren't evicted due to the lack of storage of the node? I've seen that happening before. The ops fill the local disk of the node and so kubernetes evicts them to keep the node stable.

That's interesting - I'll definitely look into that as well. We are using ephemeral storage requests and providing each node with expanded storage but it's possible we're overcommitting by not requesting the correct resource

This is the syntax we use for ephemeral storage:

extra_tags = {
        "dagster-k8s/config": {
            "pod_spec_config": {
                "volumes": [
                    {
                        "name": "scratch-data",
                        "ephemeral": {
                            "volumeClaimTemplate": {
                                "spec": {
                                    "accessModes": ["ReadWriteOnce"],
                                    "storageClassName": "ssd",
                                    "resources": {
                                        "requests": {"storage": str(size) + "Gi"}
                                    },
                                }
                            }
                        },
                    }
                ]
            },
            "container_config": {
                "volume_mounts": [{"mountPath": mount_point, "name": "scratch-data"}]
            },
        }
    }

Been looking at this a bit over the last week and have noticed a few things that have helped so far. For one thing we definitely had to set up some

PodDisruptionBudgets

to prevent the autoscaler and our k8s-descheduler from evicting dagster pods. The other was related to the disk storage symptom in that we had a few pods that were set up with too-high memory limits and were causing memory pressure on the node. Even after those two fixes we are still seeing the k8s control plane tainting the nodes that the dagster-run pod exists on with a “NoSchedule” taint. Eventually k8s will consider the node unschedulable as well and the entire node is brought down.

cc @johann re: run_monitoring feedback above

@Dagster Bot issue K8s run monitoring fails when both a run worker and step worker go down on the same node

Created issue at: https://github.com/dagster-io/dagster/issues/8183

@fahad would you be able to share (either here or in a DM) a debug file for the run? I’m curious what the step events look like

Can do! Thanks for looking into it!