Hi I m trying to run a simple pipeline with `k8s job executo

Question

Hi I m trying to run a simple pipeline with `k8s job executor` This is the pipeline code ```from dagster import solid ModeDefinition pipeline default executors fs io manager from dagster k8s import k8s job executor solid config schema= name str def hello context return f Hello context solid config name solid def bye context hello str return f Goodbye hello str split 1 MODE PROD = ModeDefinition name= prod resource defs= io manager fs io manager executor defs=default executors + k8s job executor MODE TEST = ModeDefinition name= test resource defs= pipeline mode defs= MODE PROD MODE TEST def my pipeline hello str = hello bye hello str ``` And this is the run config ```solids hello config name Dagster execution k8s config job namespace ``` The first solid `hello` succeeds but the second one `bye` fails with the following error ```Traceback most recent call last File home appuser local bin dagster line 8 in lt module gt sys exit main File home appuser local lib python3 9 site packages dagster cli init py line 48 in main cli auto envvar prefix=ENV PREFIX pylint disable=E1123 File home appuser local lib python3 9 site packages click core py line 829 in call return self main args kwargs File home appuser local lib python3 9 site packages click core py line 782 in main rv = self invoke ctx File home appuser local lib python3 9 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File home appuser local lib python3 9 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File home appuser local lib python3 9 site packages click core py line 1066 in invoke return ctx invoke self callback ctx params File home appuser local lib python3 9 site packages click core py line 610 in invoke return callback args kwargs File home appuser local lib python3 9 site packages dagster cli api py line 194 in execute step command execution plan = create execution plan File home appuser local lib python3 9 site packages dagster core execution api py line 718 in create execution plan resolved run config = ResolvedRunConfig build pipeline def run config mode=mode File home appuser local lib python3 9 site packages dagster core system config objects py line 179 in build raise DagsterInvalidConfigError dagster core errors DagsterInvalidConfigError Error in config for pipeline my pipeline Error 1 Received unexpected config entry k8s at path root execution Expected in process config marker to close String retries disabled enabled multiprocess config max concurrent Int retries disabled enabled ``` Any idea what s going on here

Yassine Marzougui · Answer

There seems to be 3 failure modes happening randomly 1 `hello` step succeeds and `bye` step fails Dagit doesn t show the error but rather ```STEP FAILURE Discovered failed Kubernetes job dagster job 0d289bfa21ffe28982593fbfc8b703a9 for step bye PIPELINE FAILURE Execution of pipeline my pipeline failed Steps failed bye ``` When inspecting the pod logs we find the error log in the message above 2 `hello` step fails with the same error above 3 The whole pipeline fails before the first step Dagit shows the following error ```An exception was thrown during execution that is likely a framework error rather than an error in user code dagster core errors DagsterInvalidConfigError Error in config for pipeline my pipeline Error 1 Received unexpected config entry k8s at path root execution Expected in process config marker to close String retries disabled enabled multiprocess config max concurrent Int retries disabled enabled Stack Trace dagster core errors DagsterInvalidConfigError Error in config for pipeline my pipeline Error 1 Received unexpected config entry k8s at path root execution Expected in process config marker to close String retries disabled enabled multiprocess config max concurrent Int retries disabled enabled File home appuser local lib python3 9 site packages dagster grpc impl py line 86 in core execute run yield from execute run iterator recon pipeline pipeline run instance File home appuser local lib python3 9 site packages dagster core execution api py line 822 in iter yield from self execution context manager prepare context File home appuser local lib python3 9 site packages dagster utils init py line 447 in generate setup events obj = next self generator File home appuser local lib python3 9 site packages dagster core execution context creation pipeline py line 354 in orchestration context event generator context creation data = create context creation data File home appuser local lib python3 9 site packages dagster core execution context creation pipeline py line 147 in create context creation data resolved run config = ResolvedRunConfig build pipeline def run config mode=pipeline run mode File home appuser local lib python3 9 site packages dagster core system config objects py line 179 in build raise DagsterInvalidConfigError ``` And sometimes the execution of step `bye` starts successfully but then fails with ```dagster core errors DagsterExecutionLoadInputError Error occurred while loading input hello str of step bye The above exception was caused by the following exception FileNotFoundError Errno 2 No such file or directory app dagster dagster home storage d6af8072 eadf 4f86 a483 af1b3e3b6472 hello result ``` Which is normal and expected here because I m using `fs io manager` with `kubernetes job executor` instead of `s3 pickle io manager`

Yassine Marzougui · Answer

Found the culprit the `pullPolicy` of the job executor was not set so it defaulted to `IfNotPresent` which caused the steps to run with an old image containing only the default executors This caused 1 and 2 The same was happening with the run worker which caused 3 but when trying to change the `pullPolicy` it turned out I m also impacted by this issue <https dagster slack com archives C01U954MEER p1630618928200600 thread ts=1630177344 082400 amp cid=C01U954MEER> This also explains the random nature of the failures because sometimes the pipeline was deployed on different nodes and the newer image was pulled