Hi wave i ve been bumping into this error occasionally using

Question

Hi wave i ve been bumping into this error occasionally using `dagster celery k8s` ```An exception was thrown during execution that is likely a framework error rather than an error in user code dagster check CheckError Invariant failed Description Pipeline run dev volumeclass pipeline 0329a1a3 4013 4dfc 8f84 d9ee13492b9e in state PipelineRunStatus STARTED expected NOT STARTED or STARTING``` any idea why this happens dagster 0 11 3

Alessandro Marrella · Answer

on a possibly related note is there a way to automatically retry the solid if this error happens

johann · Answer

Hi < Alessandro Marrella> usually this error comes up because the run worker restarted Currently we don t support retries at the pipeline level so we don t expect the run to already be started Could you confirm that the pod is named `dagster run ` the run worker as opposed to `dagster job ` the step worker

Alessandro Marrella · Answer

correct the issue is in the `run` container ``` kubectl logs dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e hbzkm n dagster class ExecuteRunArgsLoadComplete class DagsterEvent event specific data class EngineEventData error class SerializableErrorInfo cause null cls name CheckError message dagster check CheckError Invariant failed Description Pipeline run dev volumeclass pipeline 0329a1a3 4013 4dfc 8f84 d9ee13492b9e in state PipelineRunStatus STARTED expected NOT STARTED or STARTING n stack File usr local lib python3 7 site packages dagster grpc impl py line 76 in core execute run n yield from execute run iterator recon pipeline pipeline run instance n File usr local lib python3 7 site packages dagster core execution api py line 80 in execute run iterator n pipeline run pipeline name pipeline run run id pipeline run status n File usr local lib python3 7 site packages dagster check init py line 169 in invariant n CheckError Invariant failed Description desc format desc=desc n File usr local lib python3 7 site packages future utils init py line 446 in raise with traceback n raise exc with traceback traceback n marker end null marker start null metadata entries event type value ENGINE EVENT logging tags message An exception was thrown during execution that is likely a framework error rather than an error in user code pid null pipeline name dev volumeclass pipeline solid handle null step handle null step key null step kind value null class DagsterEvent event specific data null event type value PIPELINE FAILURE logging tags message This pipeline run has been marked as failed from outside the execution context pid null pipeline name dev volumeclass pipeline solid handle null step handle null step key null step kind value null ```

Alessandro Marrella · Answer

looking at kubernetes events it doesn t look the run pod was ever restarted though

johann · Answer

How is the run being launched

johann · Answer

schedule from dagit etc

Alessandro Marrella · Answer

From Dagit playground

Alessandro Marrella · Answer

This doesn t always happen and in this case it happened after a few steps not immediately

johann · Answer

For debugging I d be interested to see `kubectl describe pod dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e hbzkm n dagster` If you re ok sharing please check that it doesn t have any secrets etc

Alessandro Marrella · Answer

nothing particularly interesting there i think ``` kubectl describe pod dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e hbzkm n dagster Name dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e hbzkm Namespace dagster Priority 0 Node REDACTED Start Time Wed 07 Apr 2021 16 16 58 +0100 Labels controller uid=4b5c1c1f 0ec1 48ea a790 af6dc114be03 job name=dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e Annotations lt none gt Status Succeeded IP REDACTED IPs IP REDACTED Controlled By Job dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e Containers dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e Container ID Image REDACTED Image ID REDACTED Port lt none gt Host Port lt none gt Args dagster api execute run class ExecuteRunArgs instance ref null pipeline origin class PipelinePythonOrigin pipeline name dev volumeclass pipeline repository origin class RepositoryPythonOrigin code pointer class FileCodePointer fn name repository python file bin repository py working directory app container image REDACTED executable path usr local bin python pipeline run id 0329a1a3 4013 4dfc 8f84 d9ee13492b9e State Terminated Reason Completed Exit Code 0 Started Wed 07 Apr 2021 16 17 00 +0100 Finished Wed 07 Apr 2021 16 17 15 +0100 Ready False Restart Count 0 Environment Variables from dagster anomaly pipeline user env ConfigMap Optional false ses keys Secret Optional false Environment DAGSTER HOME opt dagster dagster home DAGSTER PG PASSWORD lt set to the key postgresql password in secret dagster postgresql secret gt Optional false DAGSTER CURRENT IMAGE REDACTED Mounts opt dagster dagster home dagster yaml from dagster instance rw path= dagster yaml var run secrets kubernetes io serviceaccount from default token 5wdps ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes dagster instance Type ConfigMap a volume populated by a ConfigMap Name dagster instance Optional false default token 5wdps Type Secret a volume populated by a Secret SecretName default token 5wdps Optional false QoS Class BestEffort Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events lt none gt ```

johann · Answer

Sorry could I get `kubectl describe job dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e n dagster` as well

Alessandro Marrella · Answer

``` kubectl describe job dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e n dagster Name dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e Namespace dagster Selector controller uid=4b5c1c1f 0ec1 48ea a790 af6dc114be03 Labels Annotations lt none gt Parallelism 1 Completions 1 Start Time Wed 07 Apr 2021 15 51 34 +0100 Completed At Wed 07 Apr 2021 16 17 16 +0100 Duration 25m Pods Statuses 0 Running 1 Succeeded 0 Failed Pod Template Labels controller uid=4b5c1c1f 0ec1 48ea a790 af6dc114be03 job name=dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e Containers dagster run 0329a1a3 4013 4dfc 8f84 d9ee13492b9e Image REDACTED Port lt none gt Host Port lt none gt Args dagster api execute run class ExecuteRunArgs instance ref null pipeline origin class PipelinePythonOrigin pipeline name dev volumeclass pipeline repository origin class RepositoryPythonOrigin code pointer class FileCodePointer fn name repository python file bin repository py working directory app container image REDACTED executable path usr local bin python pipeline run id 0329a1a3 4013 4dfc 8f84 d9ee13492b9e Environment Variables from dagster anomaly pipeline user env ConfigMap Optional false ses keys Secret Optional false Environment DAGSTER HOME opt dagster dagster home DAGSTER PG PASSWORD lt set to the key postgresql password in secret dagster postgresql secret gt Optional false DAGSTER CURRENT IMAGE REDACTED Mounts opt dagster dagster home dagster yaml from dagster instance rw path= dagster yaml Volumes dagster instance Type ConfigMap a volume populated by a ConfigMap Name dagster instance Optional false Events lt none gt ``` thanks for looking at this

johann · Answer

gt This doesn t always happen and in this case it happened after a few steps not immediately This is strange because the run worker is the process spawning steps Really seems like it would be a duplicate run worker but your manifests are showing only a single pod

johann · Answer

E g this came up with another user s custom run launcher that could create multiple run workers for the same run <https dagster slack com archives CCCR6P2UR p1606923838039400>

johann · Answer

Not sure how that s happening here

Alessandro Marrella · Answer

in my case i m using the vanilla celery k8s one is it possible it tries to spawn a new run again possibly due to some blip i m using quite small nodes and i didn t constrain resources yet so i wonder if this happens because of that