Has anyone seen this before when running a execut...
# deployment-kubernetes
Has anyone seen this before when running a execute_k8s_job? The service account and necessary permissions are definitely there and testing works but it seems that for some reason Dagster isn't picking up the "service_account_name" variable and using it..
Copy code
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '529ad498-e8f3-4871-bd58-3213cc446b09', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'd7aa5e50-269b-43a4-8925-44c8e56672e3', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'beaa185a-b8cf-4354-81a7-a5e7e28d52ba', 'Date': 'Mon, 10 Apr 2023 23:24:32 GMT', 'Content-Length': '304'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"XXXXXXXXXXX\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
service_account_name is going to configure the service account for the job that gets created, but it won't necessarily apply to the callsite that is creating the job. you may need to separately configure the place where the execute_k8s_job call is originating to have the right permissions to create jobs
Would that be via a run launcher?
Not if you're using execute_k8s_job, no
it would really depend on where the call is coming from - but it's not really something that Dagster will control
for example in the Dagster helm chart, we give this role to the dagster pods so that they have permissions to create jobs: https://github.com/dagster-io/dagster/blob/master/helm/dagster/templates/role.yaml#L1-L21
It's not coming from within K8's. We run Dgster and Dagit on an external EC2 instance.
Is system:anonymous the user that you're expecting?
i think you'll probably need to have the right credentials / role in the kubeconfig that you're using
ok, I'll look in to that further and report back if there are any more issues. Thanks!