https://dagster.io/ logo
Join Slack
Powered by
Hi Team, is it possible to deploy spark docker ima...
# ask-community
s
Hi Team, is it possible to deploy spark docker image in dagster-helm. In this example
Copy code
dagster-user-deployments:
  enabled: true
  deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "<http://docker.io/dagster/user-code-example|docker.io/dagster/user-code-example>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/example_project/example_repo/repo.py"
      port: 3030
Can i use spark image in place of docker.io/dagster/user-code-example
a
ya the “user-code-example” is intended to be replaced by your actual code
s
But is it possible to use spark-based-images which does spark-submit cmd ?
Copy code
dagster-user-deployments:
  enabled: true
  deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "<http://docker.io/bitnami/spark-3.1.2|docker.io/bitnami/spark-3.1.2>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - spark-submit --master <k8s://https>://<k8s-apiserver-host>:<k8s-apiserver-port> \
    --deploy-mode cluster
        - "/example_project/example_repo/repo.py"
      port: 3030
like this?
a
The purpose of this deployment is to run the grpc server which serves your 
@repository
and the `@job`s & etc it contains. Why are you trying to 
spark-submit
from there?
s
I need to deploy spark image in k8 which spark-submit jobs in cluster mode , so i thought using feature of dagster-k8, which allows us to submit our own images
a
I would expect a behavior like that to happen in python in a 
@resource
or 
@op
what you posted above certainly wont work since you are passing the 
spark-submit
command as a cli flag to 
dagster api grpc
in the 
dagsterApiGrpcArgs
section
s
Okay, is there any other way to achieve this ?
In dagster
And if i have op which spark-submits a job, in this case, can i build a image and submit it in this way?
Copy code
dagster-user-deployments:
  enabled: true
  deployments:
    - name: "code"
      image:
        repository: "<http://docker.io/code/code|docker.io/code/code>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/main.py"
      port: 3030
a
ya that looks like the right path - build an image with your user code and any spark dependencies and use it here as your 
user-deployment
. The same image will be used with 
@job
s are run that were sourced from that deployment.
s
Okay, which means no spark-submit is possible to pass in user-deployment. Correct me if i am wrong?
a
I do not believe it is possible to include a spark-submit command as part of the deployment spec for the user-deployment
s
And is it possible to trigger the launch-run button programmatically without using dagit UI??
a
you can set up a 
schedule
or 
sensor
or use the GraphQL API to programtically trigger runs
s
Thanks @alex
Open in Slack
PreviousNext
Powered by