https://dagster.io/ logo
#dagster-cloud
Title
# dagster-cloud
l

Lee Beeler

02/16/2024, 5:58 PM
Hello, I'm trying to run the pyspark EMR example, and I've gotten it to successfully submit to EMR when running locally through
dagster dev
. But if I run the same code from Dagster Cloud Serverless, I get the following error. It does not manage to add a step to the EMR cluster. I'm using the same IAM user locally and on Dagster Cloud.
dagster bot responded by community 1
🤖 1
z

Zach

02/16/2024, 6:10 PM
Is this failing on the Dagster side of things (not getting to the point where it fails on EMR)? I would recommend trying the LazyPySparkResource. I've ran into similar issues and have had a lot better luck that that resource when using step launchers
l

Lee Beeler

02/16/2024, 6:28 PM
it appears to work with LazyPySparkResource. thanks!
🎉 1
6 Views