https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Mykola Palamarchuk

04/29/2022, 3:07 PM
Hey team! I'm testing the behavior of
includeConfigInLaunchedRuns
in the latest version of Dagster (0.14.13). Here is my setup (using minikube on docker): •
dagster/dagster
Helm chart deployed to
dagster
k8s namespace with
dagster-user-deployments.enableSubchart=false
dagster/gagster-user-deployments
Helm chart is deployed to a separate
dpl
k8s namespace, with
includeConfigInLaunchedRuns=true
And apparently my setup does not work from the box 😁. I've investigated the reason: 1. Attempt to run a job from Dagit results in an error:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:dagster:dagster\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"dpl\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
. I've checked the code and can say, that RoleBinding is actually created in
dpl
namespace, but the subject service account name there is
dpl-dagster-user-deployments-user-deployments
instead of
dagster
. Here is the corresponding code part. I believe it must be something configurable (seting dagster k8s namespace/service-account) instead of default service account name. I was able to temporarily overcome the problem applying correct RoleBinding by hand to test it further. 2. After that job starts successfully, but it is stuck infinitely. And the reason is: few config-maps are missing in
dpl
namespace, so pods can't be created. I've copied them from
dagster
namespace to
dpl
namespace to overcome the problem. Here are the messages from `kubectl describe pods`: a.
MountVolume.SetUp failed for volume "dagster-instance" : configmap "dagster-instance" not found
b.
Error: configmap "dagster-pipeline-env" not found
After a bit of hacking described above I was able to make it work. I hope this is the whole list of the problems. @daniel, I hope my investigation helps. And thank you very much for your great work!
d

daniel

04/29/2022, 3:09 PM
Hey Mykola - it's correct that a bit of extra config is needed in order for the dagster-user-deployments subchart to work in another namespace. You'll need to create those configmaps manually and give the serviceaccount some additional permissions so that it can launch jobs in other namespaces too. Glad you were able to sort it out, we'll see what we can to to make this more streamlined going forward
m

Mykola Palamarchuk

04/29/2022, 3:23 PM
My way to overcome the problems is not a solution at all. I did that just to investigate the problem and help you somehow. I can't perform the same tricks with CD tool, so for the moment I will still use "duplicate configmaps" solution instead of
includeConfigInLaunchedRuns
based one. But I'm really interested in it. So I'll try to help you with testing.
d

daniel

04/29/2022, 3:24 PM
Got it - so you're currently unable to use the helm chart with the user code deployments in a different namespace?
using includeConfigInLaunchedRuns?
m

Mykola Palamarchuk

04/29/2022, 4:36 PM
Yes, exactly. The problem #1 looks like a bug, but I can add some static RoleBinding to overcome it. But #2 is an actual blocker as I can't effectively generate required config maps from dagster namespace in deployments namespace. I could copy them by hand from one namespace to another, but that's not the case for CI/CD.
d

daniel

04/30/2022, 1:05 AM
I believe https://github.com/dagster-io/dagster/pull/7660 will fix #2 - we should be able to get that out next week
#1 is a little trickier to handle cleanly in a way that covers all the different ways that people might use the two charts
m

Mykola Palamarchuk

05/02/2022, 9:12 AM
I was also thinking about hypothetical possibilities to run multiple instances of Dagster in different namespaces connected to one deployments. I'm not sure what may be the motivation to do like that. So may be the best option would be to consider using one dagster per deployments for now, which looks easy to implement.
12 Views