Hi Team is it possible to deploy spark docker image in dagst

Question

Hi Team is it possible to deploy spark docker image in dagster helm In this example ```dagster user deployments enabled true deployments name k8s example user code 1 image repository tag latest pullPolicy Always dagsterApiGrpcArgs python file example project example repo repo py port 3030``` Can i use spark image in place of

alex · Answer

ya the user code example is intended to be replaced by your actual code

Sundara Moorthy · Answer

But is it possible to use spark based images which does spark submit cmd

Sundara Moorthy · Answer

```dagster user deployments enabled true deployments name k8s example user code 1 image repository tag latest pullPolicy Always dagsterApiGrpcArgs spark submit master lt k8s apiserver host gt lt k8s apiserver port gt deploy mode cluster example project example repo repo py port 3030```

Sundara Moorthy · Answer

like this

alex · Answer

The purpose of this deployment is to run the grpc server which serves your ` repository` and the ` job`s amp etc it contains Why are you trying to `spark submit` from there

Sundara Moorthy · Answer

I need to deploy spark image in k8 which spark submit jobs in cluster mode so i thought using feature of dagster k8 which allows us to submit our own images

alex · Answer

I would expect a behavior like that to happen in python in a ` resource` or ` op` what you posted above certainly wont work since you are passing the `spark submit` command as a cli flag to `dagster api grpc` in the `dagsterApiGrpcArgs` section

Sundara Moorthy · Answer

Okay is there any other way to achieve this

Sundara Moorthy · Answer

In dagster

Sundara Moorthy · Answer

And if i have op which spark submits a job in this case can i build a image and submit it in this way ```dagster user deployments enabled true deployments name code image repository <http docker io code code|docker io code code> tag latest pullPolicy Always dagsterApiGrpcArgs python file main py port 3030```

alex · Answer

ya that looks like the right path build an image with your user code and any spark dependencies and use it here as your `user deployment` The same image will be used with ` job` s are run that were sourced from that deployment

Sundara Moorthy · Answer

Okay which means no spark submit is possible to pass in user deployment Correct me if i am wrong

alex · Answer

I do not believe it is possible to include a spark submit command as part of the deployment spec for the user deployment

Sundara Moorthy · Answer

And is it possible to trigger the launch run button programmatically without using dagit UI

alex · Answer

you can set up a `schedule` or `sensor` or use the GraphQL API to programtically trigger runs

Sundara Moorthy · Answer

Thanks < alex>