Hey Team, I have docker image with dagster install...
# ask-community
s
Hey Team, I have docker image with dagster installed to it. And i am able to deploy it to k8 as well with scaling. So what's the different between this deployment vs dagster-k8-deployment (which has user-code-deployment). In my understanding dagster-k8-deployment will run each step as different k8 jobs, so it can handle the huge load. And in case of normal image (with dagster installed) will run in default runlauncher mode which can't handle huge request load. Example: Assume that we have 2 k8 pods with 4 cores in our normal deployment and i have submitted 3 request (each request will have a graph of 4 ops which can run simultaneously) to it, so in this case we have around 12 jobs, so at any given point of time we will be able to run only 8 jobs parallelly using this deployment. Remaining 4 jobs have to wait. But this is not the case if we use dagster-k8 deployment as it will create new k8 pod for each run & step. Pls correct me if i am wrong?
d
Hi Sundara -The K8sRunLauncher will launch each run in its own K8s pod, that's the default if you use the dagster helm chart in kubernetes. You can optionally also turn on the k8s_job_executor for some or all of your jobs, which will run each step in its own k8s pod as well. And yeah, the auto-scaling capabilities of k8s will make it easier to scale up your workload and run the jobs and steps on multiple machines
s
Copy code
Assume that we have 2 k8 pods with 4 cores in our normal deployment and i have submitted 3 request (each request will have a graph of 4 ops which can run simultaneously) to it, so in this case we have around 12 jobs, so at any given point of time we will be able to run only 8 jobs parallelly using this deployment. Remaining 4 jobs have to wait.
is my understanding correct for the above normal deployment with normal python image with dagster installed to ?
d
If you're running Dagster in a single image without the k8s run launcher or the helm chart, it will run everything in a single pod, so it will only have access to whatever resources the cluster has allowed for that single pod.
so i think it's worth than that - we recommend using the helm chart if possible
s
what would be the behavior in case we have 2 dagster pods? will it be able to split & maintain the load?
Currently we are trying to achieve dynamic graph in repositories which is not available currently.
d
it won't - i would use the helm chart if possible
s
Okay will try to use helm charts, in that case is there any way i can add the user-code-repository dynamically?
Example: i got a request based on which i need to create the repository and add it to the dagster, so it can triggered.
d
right now it requires a helm upgrade when the code changes. There's a feature request issue for taking more of a service discovery approach to this here: https://github.com/dagster-io/dagster/issues/6295
s
Okk, but in our usecase we will not update the code-base. We will add the new repository, so it can triggered using graphql api's. I hope in this case we just need to update the workspace.yml. Pls correct me if i am wrong?
d
The helm chart creates the workspace.yaml for you - any time you want to update the workspace.yaml you'll need to run a helm upgrade. If you just want to restart a user code deployment without changing the image that it's running you could stop the pod via kubectl and Kubernetes will restart it for you
Not sure if the dagster cloud product is something you might be interested but that has a bit more functionality for updating your code via API without needing to run any helm commands
While still running your code in Kubernetes