https://dagster.io/ logo
Powered by
#deployment-kubernetes
Title
# deployment-kubernetes
r

Rubén Lopez Lozoya

03/18/2021, 7:47 PM
Hey, can I assign a service account to the k8s jobs so that they can use GCP Workload Identity? If so, how can I do it? (I'm using the Helm chart)
r

rex

03/18/2021, 8:07 PM
we do offer the ability to specify the service account name for the K8sRunLauncher but looks like it’s currently not configurable (in Helm, it’s set to be the service account generated from our chart)
wanna file a feature request for this?
r

Rubén Lopez Lozoya

03/18/2021, 8:32 PM
yeah that would be awesome, how can I allow then access to, for example, Google Cloud Storage from the jobs? so that they can use it for intermediate storage
I have to provide it in the pipeline config yaml?
n

Noah K

03/18/2021, 8:45 PM
As you said, you need an annotation in the service account
And then some GCP IAM rules to authorize it
r

Rubén Lopez Lozoya

03/18/2021, 8:56 PM
i havent work with annotations before, any useful resource you can point me to?
n

Noah K

03/18/2021, 9:02 PM
And you also need to set up an iam policy binding on the GCP side https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to
Basically full bidirectional linkage, the K8s service account points to which GCP service account it matches with, and the GCP SA lists K8s SAs that are allowed to match it.
Once you do that, anything using the default google SDK tooling will pick up that GCP SA's permissions automatically
r

Rubén Lopez Lozoya

03/18/2021, 9:11 PM
The systems engineer already created the workload identity, and he told me I must point to the GCP SA from my K8s job (the one launched by dagster). The thing is that he told me I need to point to it from the job definition yaml, which I cannot configure since it is generated by the K8sJobLauncher, hence my question. So, I believe everything you've just said has been done by him, and the last point is how to tell the K8sJobLauncher to launch the jobs pointing to that service account. I see in the link you sent that when defining the job with a yaml, you specifiy 
serviceAccountName
as part of the job spec, but I am unable to define the Dagster jobs myself, so I have to tell the job launcher to do it, and that is the original question, where do I tell him to do so. 😞
Do you mean that I should bind GCP permissions to the serviceAccount generated by Dagster Helm chart?
n

Noah K

03/18/2021, 9:19 PM
Right, the issue is you need to patch in the annotation I mentioned
You can use a post render filter to add the annotation
(or fork and edit the chart)
m

Matyas Tamas

03/18/2021, 9:20 PM
another thing you can do is associate to the cluster's default service account
n

Noah K

03/18/2021, 9:21 PM
@Matyas Tamas No, as I linked there is a specific SA created by dagster
r

Rubén Lopez Lozoya

03/18/2021, 9:21 PM
that last thing would mean setting undesired permissions by default
to the cluster
n

Noah K

03/18/2021, 9:21 PM
That's the SA created for you
But as @rex mentioned, it doesn't allow setting annotations via chart values
Hence a post render overlay 🙂
r

Rubén Lopez Lozoya

03/18/2021, 9:22 PM
Cool, very well explained guys, thanks a lot! You are really helping me a lot with the k8s thing these days!
n

Noah K

03/18/2021, 9:23 PM
Also your system engineer will need to know the name (and namespace) of your K8s service account to set up the policy binding I mentioned/linked
m

Matyas Tamas

03/18/2021, 9:23 PM
@Rubén Lopez Lozoya its true that that will be more permissive than you might want. @Noah K there is a service account created by dagster. There is also 
<project_id>.svc.id.goog[default/default]
which will work as well.
n

Noah K

03/18/2021, 9:24 PM
@Matyas Tamas WorkloadID only works with the K8s SA the pod is actually running with
So no, that will not work
m

Matyas Tamas

03/18/2021, 9:24 PM
hmm - I have it running...
n

Noah K

03/18/2021, 9:24 PM
You might be used to the default case when you do not specify a 
serviceAccountName
That does use the 
default
SA then 🙂
m

Matyas Tamas

03/18/2021, 9:25 PM
correct - which I believe is the default, no?
ah - I see
n

Noah K

03/18/2021, 9:26 PM
The dagster SA gets threaded through in the config
m

Matyas Tamas

03/18/2021, 9:31 PM
wait, so how are you setting the serviceAccountName?
in every job config?
n

Noah K

03/18/2021, 9:33 PM
Not sure I understand the question. It's a config option in the dagster_k8s plugin
m

Matyas Tamas

03/18/2021, 9:33 PM
yup
n

Noah K

03/18/2021, 9:33 PM
When is then passed down into the Kubernetes jobs it creates
m

Matyas Tamas

03/18/2021, 9:39 PM
like as a tag you mean? i.e.:
Copy code
@solid(tags={
    "dagster-k8s/config": {
        "job_config": {
            "service_account_name": "<service account>"
        }
    }
})
w

William Sheu

03/18/2021, 9:39 PM
with the k8s celery executor, you can just specify it in the run config
m

Matyas Tamas

03/18/2021, 9:39 PM
ah - right
w

William Sheu

03/18/2021, 9:39 PM
Copy code
execution:
  celery-k8s:
    config:
      ...
      service_account_name: ...
m

Matyas Tamas

03/18/2021, 9:40 PM
cool! thanks all!
n

Noah K

03/18/2021, 9:42 PM
I'm not aware of any support for it in tags but maybe it's at some meta level I don't know about 🙂
Would be nice though
2 Views
Powered by