Has anyone seen this before when running a execute_k8s_job? The service account and necessary permissions are definitely there and testing works but it seems that for some reason Dagster isn't picking up the "service_account_name" variable and using it..

kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '529ad498-e8f3-4871-bd58-3213cc446b09', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'd7aa5e50-269b-43a4-8925-44c8e56672e3', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'beaa185a-b8cf-4354-81a7-a5e7e28d52ba', 'Date': 'Mon, 10 Apr 2023 23:24:32 GMT', 'Content-Length': '304'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:anonymous\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"XXXXXXXXXXX\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

service_account_name is going to configure the service account for the job that gets created, but it won't necessarily apply to the callsite that is creating the job. you may need to separately configure the place where the execute_k8s_job call is originating to have the right permissions to create jobs

Would that be via a run launcher?

Not if you're using execute_k8s_job, no

It's not coming from within K8's. We run Dgster and Dagit on an external EC2 instance.

Is system:anonymous the user that you're expecting?

ok, I'll look in to that further and report back if there are any more issues. Thanks!