https://dagster.io/ logo
Join Slack
Powered by
Has anyone looked into running Dagster user code d...
# deployment-kubernetes
m
Has anyone looked into running Dagster user code deployments in multiple alternate namespaces? I just tried that, and ran into a blocker and an interesting observation. 🧵
So to describe the scenario first of all. I had the Dagster Helm chart (Dagit, Daemon) deployed into the 
dagster
namespace. Then, a Foo user code deployment deployed in 
foo
, and a Bar one in the 
bar
namespace.
The first problem I ran into was that the user code deployments relied on the 
dagster-postgresql-secret
secret, as seen here
But I provisioned one of those in the 
foo
and 
bar
namespaces (with garbage values) and the whole system was happy. I don't know if the garbage password would pose a problem, since I didn't get runs running, but it did make me wonder if this secret is actually needed in the subchart?
So then the blocker I ran into was the Dagster k8s run launcher: it only supports a single job namespace. It defaults to the release namespace (in this case 
dagster
) or you can override it -- but you can't set it to a list.
Oh, another issue I ran into was the Dagster service account's bound role. It only grants permission to operate in the run launcher's job namespace. So you'd need either a ClusterRole (scary) or a bunch of different per-namespace permissions I guess.
a
Regarding the psql secret. A solution could be to modify the helm chart to allow setting annotations on the secret ( here ) and then use https://github.com/emberstack/kubernetes-reflector to mirror the secret across different namespaces
m
Yep that's certainly true. We tend to shy away from that strategy just out of an interest in having a little more control over what we're creating and where, but it's a valid option. I think if I were going to continue down this path, I would be able to just inject the Postgres secret as a K8s manifest alongside the Helm release. A little more inconvenient to be sure.
👍 1
a
Regarding the dagsterk8slauncher. I think it would be nice if by default the jobs are run in the same namespace where the user-code lives.
👍 2
but i belive it’s not possible at it’s current state
m
Yeah -- I took a look through the code to see how difficult it would be to lift the variable into a list, and it's everywhere unfortunately.
I am not bold enough to take that on as a PR at this time haha 😅
a
If ever I have a project that is implementing this kind of configuration: https://github.com/konpyutaika/data-platform/tree/main/EKS • Namespace 
system-dagster
: with one daemon and one DagIt instance that sees all user deployments. • Namespace 
business
: with some user deployments + a DagIt instance that only sees user deployments of its namespace. • Namespace `product`: with some user deployments + a DagIt instance that only sees the user deployments of its namespace And whatever DagIt instance you run the job from for run and steps are executed in the same namespace as the user deployment containing the code definition and obviously with the same SA as the user deployment. (the related architecture is described here)
m
Oh interesting. The "system" Dagit in `system-dagster`: are you able to schedule jobs in 
business
and 
product
from that Dagit? Or is it a read-only Dagit?
a
There is only one daemon that is responsible for scheduling tasks associated with that the system dagit for all jobs. And the Dagit per namespace are not read-only but I don't deploy a daemon. This way, only the system-dagit manages the trigger rules, and per-namespace Dagits still allow teams to run tasks manually! And if I run the job from the scheduler, system dagIt or dagIt by namespace, it doesn't change the location of the runtime pod and steps, it's still in the same namespace as the user deployment. (This isn't a production setup yet, and maybe I'm missing something, but from all the testing I've done so far, it seems to work).
👀 3
a
Interesting setup… Let us know if you end up deploying it in prod
a
I will, as soon as our POC convinces us that this is the right plan to let the Airflow for dagster 😛
5 Views
Open in Slack
PreviousNext
Powered by