Afternoon All Running into some very weird issues with our E

Question

Afternoon All Running into some very weird issues with our EMR serverless deployment that I can t quite manage to reproduce elsewhere We had a materialization run that uses the EMR Step launcher however ended up being marked incorrectly as failed due to a boto exception After that any runs ran against it end up failing due to another reason Is this cause for concern or is it the type of thing that should self resolve after some of the runners restart Forgive me if I m not completely correct with the term Details in thread thread UPDATE Waited awhile before rerunning and it seems like it s resolved Still a very strange issue slightly smiling face

Zach P · Answer

First Job Failed with ```dagster aws emr emr EmrError EMR log file did not appear on S3 after waiting File usr local lib python3 8 site packages dagster core execution plan execute plan py line 224 in dagster event sequence for step for step event in check generator step events File usr local lib python3 8 site packages dagster aws emr pyspark step launcher py line 310 in launch step yield from self wait for completion and log run id step key emr step id step context File usr local lib python3 8 site packages dagster aws emr pyspark step launcher py line 323 in wait for completion and log self log logs from s3 step context log emr step id File opt dagster app common clusters py line 165 in log logs from s3 stdout log stderr log = self emr job runner retrieve logs for step id File usr local lib python3 8 site packages dagster aws emr emr py line 379 in retrieve logs for step id stdout log = self wait for log log log bucket prefix stdout gz format prefix=prefix File usr local lib python3 8 site packages dagster aws emr emr py line 419 in wait for log raise EmrError EMR log file did not appear on S3 after waiting from err The above exception was caused by the following exception botocore exceptions WaiterError Waiter ObjectExists failed An error occurred 403 Forbidden File usr local lib python3 8 site packages dagster aws emr emr py line 413 in wait for log waiter wait File usr local lib python3 8 site packages botocore waiter py line 55 in wait Waiter wait self kwargs File usr local lib python3 8 site packages botocore waiter py line 357 in wait raise WaiterError ``` The above error may be a one time issue iam etc I haven t seen it happen before but I m more concerned with the invalid state in may have introduced caused I now have both STEP SUCCESS and STEP FAILURE in the logs

Zach P · Answer

Now when I run it again to refresh the partition just in an attempt to remove the ambiguity I get an Invariant failed error ```dagster check CheckError Invariant failed Description Attempted to mark step poi scores as complete that was not known to be in flight File usr local lib python3 8 site packages dagster core execution api py line 1085 in pipeline execution iterator for event in pipeline context executor execute pipeline context execution plan File usr local lib python3 8 site packages dagster core executor multiprocess py line 219 in execute active execution handle event event or none File usr local lib python3 8 site packages dagster core execution plan active py line 400 in handle event self mark failed step key File usr local lib python3 8 site packages dagster core execution plan active py line 342 in mark failed self mark complete step key File usr local lib python3 8 site packages dagster core execution plan active py line 387 in mark complete check invariant File usr local lib python3 8 site packages dagster check init py line 1470 in invariant raise CheckError f Invariant failed Description desc ```

daniel · Answer

Hey Zach I ll go ahead and file an issue for this as it looks like something the EmrPySparkStepLauncher should be able to recover from better It looks like what happened was that the EMR step completed hence the success event but then trying to get the EMR logs from S3 failed and created a STEP FAILURE event failing with a 403 Forbidden is somewhat surprising I m not actually totally sure if that s an expected failure where we should keep retrying or not the intention is that it waits for a while until the object exists but it s possible the 403 is short circuiting that Do you have any idea why it would be 403ing with a permissions issue rather than telling you that the object doesn t exist The relevant code that we re hitting is here <https github com dagster io dagster blob master python modules libraries dagster aws dagster aws emr emr py L404 L419> Separately if you don t actually care about these logs you could set wait for logs to False but that s the default so i imagine you set it to True intentionally

daniel · Answer

From our logs I think you actually hit the same invariant in both runs it s getting confused by the STEP SUCCESS then STEP FAILURE thing but it s kind of a secondary problem the main problem is that 403

daniel · Answer

Ah OK here s a theory <https github com aws aws sdk java issues 974> java not python but same issue suggests that this may go away if the IAM role you re using has the s3 List permission

Zach P · Answer

Hmm interesting I m not entirely sure why a 403 is occurring here I haven t made any changes to the workflow here and running it again seems to resolve the 403 The dagster role we re running with has List permissions on the bucket but not List for all buckets Ironically actually noticed some were missing but not the buckets that would be used here One thing that may be worth noting is that I m actually subclassing the PysparkStepLauncher to add some extra properties and an override for the ` log logs from s3` method It s almost the same just an if else statement in the record logging loop

daniel · Answer

it s possible that when it worked you were lucky and the bucket exists by the first time that it checked

Zach P · Answer

ahhhh

daniel · Answer

If you have s3 ListBucket permissions on the bucket in question I wouldn t expect it to raise a 403 though

daniel · Answer

based on the Permissions section of <https boto3 amazonaws com v1 documentation api latest reference services s3 html highlight=query S3 Client head object>

daniel · Answer

I m not sure why the Waiter doesn t give you a mode where it also retries on 403s if it did we could use it here

Zach P · Answer

Agreed Many things I wish boto3 had or did differently I ll check the IAM again in the morning