Hello, I'm trying to run the pyspark EMR example, and I've gotten it to successfully submit to EMR when running locally through

dagster dev

. But if I run the same code from Dagster Cloud Serverless, I get the following error. It does not manage to add a step to the EMR cluster. I'm using the same IAM user locally and on Dagster Cloud.

Is this failing on the Dagster side of things (not getting to the point where it fails on EMR)? I would recommend trying the LazyPySparkResource. I've ran into similar issues and have had a lot better luck that that resource when using step launchers

it appears to work with LazyPySparkResource. thanks!