Hi I m trying to archive dagster and its dependencies as a z

Question

Hi I m trying to archive dagster and its dependencies as a zip so I can pass it as a `spark submit` parameter to run it on a cluster mode but now I m getting this error when the it s trying to import dagster ```from dagster import File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster init py line 110 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster core launcher init py line 2 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster core launcher cli api run launcher py line 6 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster api execute run py line 2 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster cli init py line 8 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster cli api py line 13 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster cli workspace init py line 1 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster cli workspace cli target py line 13 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster core host representation init py line 47 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster core host representation repository location py line 8 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster api snapshot execution plan py line 9 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster grpc init py line 11 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip dagster grpc client py line 7 in lt module gt File appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip grpc init py line 23 in lt module gt ImportError cannot import name cygrpc from grpc cython appdata hdfs v8 yarn nm usercache hismail appcache application 1602600045749 421754 container e168 1602600045749 421754 01 000009 dependencies zip grpc cython init py ``` any idea I have tried to include `grpcio` separately from `dagster` as well but the issue still persists

sandy · Answer

I haven t tried to distribute dagster via spark submit before Unfortunately when libraries depend on native code like dagster with grpc I think it s generally difficult to distribute them through a python zip Is it possible for you to install dagster on the cluster Does dagster itself need to be on the cluster Are you using a step launcher

Haydar Ali Ismail · Answer

We are trying to avoid installing the `dagster` itself on the cluster to make the solution as portable as much as we can I think I do need to have `dagster` because when I try to run the pipeline it complains that it cannot import `dagster` I am actually trying to make my own step launcher for a self hosted Spark but having a bit of an issue trying to mimic the step and events stuff based from the Amazon EMR pyspark step launcher To make a bare minimum step launcher what are the stuff that has to be implemented

sandy · Answer

Apologies for the delay here Haydar First off I filed to track the packaging issue you reported I also filed to track potentially adding a step launcher that can launch to an on prem yarn cluster Last I wrote up some instructions on how to run a step launcher It s not easy disappointed But deploying code remotely is never very easy Another option would be to invoke spark submit from within the body of your solid