Hi everyone again a question with dagster pyspark for which

Question

Hi everyone again a question with dagster pyspark for which I tried to look for previous conversations but didn t find any solution I am trying to use emr pyspark step launcher for my emr cluster and to pass pipeline path I am doing ` local pipeline package path str Path file parents 1 ` so that code zip can have parent of parent directory as I have some utility files which I would need to run with pyspark However my run fails with `ModuleNotFoundError` does anyone have any idea or have faced such issue Thanks

Zach P · Answer

Where does the fail happen If you re using the EMR launcher it could fail either while trying to launch the job from the dagster runner or in the EMR step job itself

Pragna · Answer

it fails in EMR step

Zach P · Answer

What s the exact module not found error When I got the EMR step launcher set up I had to install many dependencies

Zach P · Answer

``` bin bash sudo yum install y python3 devel sudo python3 m pip install dagster dagster aws boto3 dagster spark sudo python3 m pip install dagster pyspark no deps``` Here s a snippet from the bootstrap script that I use with EMR not 100 if this is what is happening to you but hope it helps slightly smiling face

Pragna · Answer

I have installed all the dependencies This error is for my utility method which I am importing from local path

Pragna · Answer

do you think I should also add util files to EMR cluster

Zach P · Answer

Yep I was just trying to find how I set mine up

Zach P · Answer

I don t have the best setup for it ATM but it essentially posts the entire repo into EMR for me

Zach P · Answer

And the repo is like ```utils generic dagster repo a dagster repo b ``` Eventually I ll need to change this to only be ```utils generic dagster repo a ```

Pragna · Answer

Yeah i want to post the entire repo right now it is just putting the pipeline file It would be great to learn about how to achieve this

Pragna · Answer

Do I need to change anything for this argument ` local pipeline package path ` whenever you have some time to check the setup please let me know Thank you for your help I really appreciate this

Zach P · Answer

I have mine set up to point to `str Path file parent parent `

Pragna · Answer

I thought `str Path file parents 1 ` will do the same thing but I will try parent parent also

Pragna · Answer

do you also set ` local job package path `

Zach P · Answer

I think those settings aren t compatible with one another I do have `deploy local pipeline package` set to `True` however

Zach P · Answer

I m not sure what the exact outcome would be TBH and it probably depends heavily on how you have your repo structured One thing that s pretty great about dagster is the ability to run it locally you can run dagit via VSCODE debugger and get breakpoints with PDB to figure out what exact values are being passed and other great stuff

Pragna · Answer

That s helpful I will try that Thank you so much for your time and help raised hands thank you box

Zach P · Answer

No worries Hope you re able to get it working shortly slightly smiling face