https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
m

Michel Rouly

02/11/2023, 9:18 PM
Has anyone looked into running Dagster user code deployments in multiple alternate namespaces? I just tried that, and ran into a blocker and an interesting observation. 🧵
So to describe the scenario first of all. I had the Dagster Helm chart (Dagit, Daemon) deployed into the
dagster
namespace. Then, a Foo user code deployment deployed in
foo
, and a Bar one in the
bar
namespace.
The first problem I ran into was that the user code deployments relied on the
dagster-postgresql-secret
secret, as seen here
But I provisioned one of those in the
foo
and
bar
namespaces (with garbage values) and the whole system was happy. I don't know if the garbage password would pose a problem, since I didn't get runs running, but it did make me wonder if this secret is actually needed in the subchart?
So then the blocker I ran into was the Dagster k8s run launcher: it only supports a single job namespace. It defaults to the release namespace (in this case
dagster
) or you can override it -- but you can't set it to a list.
Oh, another issue I ran into was the Dagster service account's bound role. It only grants permission to operate in the run launcher's job namespace. So you'd need either a ClusterRole (scary) or a bunch of different per-namespace permissions I guess.
a

Andrea Giardini

02/11/2023, 9:27 PM
Regarding the psql secret. A solution could be to modify the helm chart to allow setting annotations on the secret ( here ) and then use https://github.com/emberstack/kubernetes-reflector to mirror the secret across different namespaces
m

Michel Rouly

02/11/2023, 9:29 PM
Yep that's certainly true. We tend to shy away from that strategy just out of an interest in having a little more control over what we're creating and where, but it's a valid option. I think if I were going to continue down this path, I would be able to just inject the Postgres secret as a K8s manifest alongside the Helm release. A little more inconvenient to be sure.
👍 1
a

Andrea Giardini

02/11/2023, 9:31 PM
Regarding the dagsterk8slauncher. I think it would be nice if by default the jobs are run in the same namespace where the user-code lives.
👍 2
but i belive it’s not possible at it’s current state
m

Michel Rouly

02/11/2023, 9:31 PM
Yeah -- I took a look through the code to see how difficult it would be to lift the variable into a list, and it's everywhere unfortunately.
I am not bold enough to take that on as a PR at this time haha 😅
a

Alexandre Guitton

02/11/2023, 10:31 PM
If ever I have a project that is implementing this kind of configuration: https://github.com/konpyutaika/data-platform/tree/main/EKS • Namespace
system-dagster
: with one daemon and one DagIt instance that sees all user deployments. • Namespace
business
: with some user deployments + a DagIt instance that only sees user deployments of its namespace. • Namespace `product`: with some user deployments + a DagIt instance that only sees the user deployments of its namespace And whatever DagIt instance you run the job from for run and steps are executed in the same namespace as the user deployment containing the code definition and obviously with the same SA as the user deployment. (the related architecture is described here)
m

Michel Rouly

02/11/2023, 10:32 PM
Oh interesting. The "system" Dagit in `system-dagster`: are you able to schedule jobs in
business
and
product
from that Dagit? Or is it a read-only Dagit?
a

Alexandre Guitton

02/11/2023, 10:41 PM
There is only one daemon that is responsible for scheduling tasks associated with that the system dagit for all jobs. And the Dagit per namespace are not read-only but I don't deploy a daemon. This way, only the system-dagit manages the trigger rules, and per-namespace Dagits still allow teams to run tasks manually! And if I run the job from the scheduler, system dagIt or dagIt by namespace, it doesn't change the location of the runtime pod and steps, it's still in the same namespace as the user deployment. (This isn't a production setup yet, and maybe I'm missing something, but from all the testing I've done so far, it seems to work).
👀 3
a

Andrea Giardini

02/12/2023, 6:02 PM
Interesting setup… Let us know if you end up deploying it in prod
a

Alexandre Guitton

02/12/2023, 7:26 PM
I will, as soon as our POC convinces us that this is the right plan to let the Airflow for dagster 😛
3 Views