Does anyone else have an issue where the kubernetes (GKE) jobs aren't getting the right service account? Its using the default service account which doesn't have access to the IO manager bucket, so my jobs keep failing. From the logs:

google.api_core.exceptions.Forbidden: 403 GET <https://storage.googleapis.com/storage/v1/b/dagster-io-manager-artifacts?fields=name&prettyPrint=false>: <mailto:1045208284552-compute@developer.gserviceaccount.com|1045208284552-compute@developer.gserviceaccount.com> does not have storage.buckets.get access to the Google Cloud Storage bucket. 
  File "/opt/conda/lib/python3.8/site-packages/dagster/core/errors.py", line 184, in user_code_error_boundary
    yield
  File "/opt/conda/lib/python3.8/site-packages/dagster/core/execution/resources_init.py", line 310, in single_resource_event_generator
    resource_def.resource_fn(context)
  File "/opt/conda/lib/python3.8/site-packages/dagster_gcp/gcs/io_manager.py", line 121, in gcs_pickle_io_manager
    pickled_io_manager = PickledObjectGCSIOManager(
  File "/opt/conda/lib/python3.8/site-packages/dagster_gcp/gcs/io_manager.py", line 21, in __init__
    check.invariant(self.bucket_obj.exists())
  File "/opt/conda/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 822, in exists
    client._get_resource(
  File "/opt/conda/lib/python3.8/site-packages/google/cloud/storage/client.py", line 349, in _get_resource
    return self._connection.api_request(
  File "/opt/conda/lib/python3.8/site-packages/google/cloud/storage/_http.py", line 80, in api_request
    return call()
  File "/opt/conda/lib/python3.8/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/opt/conda/lib/python3.8/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/opt/conda/lib/python3.8/site-packages/google/cloud/_http/__init__.py", line 494, in api_request
    raise exceptions.from_http_response(response)

Looking at the job itself seems to demonstrate the

serviceAccount

is correct but the managed fields is not:

managedFields:
  - apiVersion: batch/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:template:
          f:spec:
            f:serviceAccount: {} # 👈 
            f:serviceAccountName: {}
# ...snip...
spec:
  template:
    metadata:
      annotations:
        <http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: "false"
      creationTimestamp: null
      labels:
    spec:
      containers:
      - image: my-cool-dagster-repo
        imagePullPolicy: Always
        name: dagster
      restartPolicy: Never
      serviceAccount: dagster # 👈
      serviceAccountName: dagster

And, in fact, the

helm.yaml

has the right service account 🤔

---
global:
  postgresqlSecretName: "dagster-postgresql-secret"
  dagsterHome: "/opt/dagster/dagster_home"
  # A service account name to use for this chart and all subcharts. If this is set, then
  # dagster subcharts will not create their own service accounts.
  serviceAccountName: "dagster"

I wonder if this has to do with upgrading to dagster 0.15? I'm not quite certain, but the problem seems to have started after I upgraded.

Hi Jeremy - if it happened starting in 0.15.0, it may have something to do with the includeConfigInLaunchedRuns flag that was moved from defaulting to false to defaulting to true: https://docs.dagster.io/changelog#extension-libraries You could try changing it to false in your user code deployment and see if that makes it start working again - would be somewhat surprising but not impossible

Huh, interesting! I'll give this a shot when I get back to my keyboard

I had this happen with my move to 0.15. I am using a separate user code deployment helm chart and had to add a line to use a global service account: https://github.com/xmarcosx/dagster-via-kubernetes/blob/master/user_code_values.yaml#L2

@marcos was the user code deployment helm chart running in a different namespace? I would have thought that the default service account that the user code deployment chart makes would still work for the launched run

Same namespace for both. I had a few Dagster deployments on GKE. I will upgrade a different one to 0.15.2 and see if it breaks in the same way.

huh. did you hit the same managedFields error that jeremy is describing here?

I hit the same issue where my gcs io manager started failing when trying to connect to the bucket. The yaml config was showing that the jobs created by Dagster were not using the global service account that I configured in my dagster helm chart. I hadn’t needed to also include the same global service account override in my user code deployment helm config until after the upgrade to 0.15

what gave the dagster service account access to the io manager though? was that something that you previously configured?

In my GCP project I have a dagster service account. That has access to the bucket. Then in GKE I deploy Dagster and tell it to use a GKE service account dagster. Finally, I bind them together:

gcloud iam service-accounts add-iam-policy-binding \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:$GOOGLE_CLOUD_PROJECT.svc.id.goog[default/dagster]" \
  dagster@$<http://GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com|GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com>;

kubectl annotate serviceaccount \
  dagster \
  <http://iam.gke.io/gcp-service-account=dagster@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com;|iam.gke.io/gcp-service-account=dagster@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com;>

that's it exactly. OK, that sounds, while annoying, like an expected consequence of the breaking change. Apologies for the trouble

Perfect. Thank you for clarifying.

Ok, I added that field to the helm values and it did nothing. But rolling back to dagster 0.14.20 fixed it! I'm not sure if I understand the problem well enough: I'm not using a separate user code deployment helm. Should I regenerate the helm values?

If making your user deployments look something like this:

dagster-user-deployments:
  enabled: true
  deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "<http://docker.io/dagster/user-code-example|docker.io/dagster/user-code-example>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/example_project/example_repo/repo.py"
      port: 3030
      includeConfigInLaunchedRuns:
        enabled: false

didn't fix it then i'm not totally sure what's going on - changing the default there from false to true was the only significant change to the helm chart that I'm aware of between 0.14.20 and 0.15.0

Yep, here's the full section:

dagster-user-deployments:
  enabled: true
  enableSubchart: true
  deployments:
    - name: "dg-workspace"
      image:
        repository: "<http://gcr.io/foo/dagster|gcr.io/foo/dagster>"
        tag: "1.0.62"
        pullPolicy: Always
      envSecrets:
        - name: run-worker-secrets
        - name: slack-creds
      annotations: {}
      nodeSelector:
        <http://iam.gke.io/gke-metadata-server-enabled|iam.gke.io/gke-metadata-server-enabled>: "true"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "api"
                operator: "In"
                values: ["yes"]
      tolerations:
      - key: "api"
        operator: "Equal"
        value: "yes"
        effect: "NoSchedule"
      podSecurityContext: {}
      securityContext: {}
      resources: {}
      includeConfigInLaunchedRuns:
        enabled: false  # 👈
      replicaCount: 1
      livenessProbe:
        initialDelaySeconds: 0
        periodSeconds: 20
        timeoutSeconds: 3
        successThreshold: 1
        failureThreshold: 3
      startupProbe:
        enabled: true
        initialDelaySeconds: 0
        periodSeconds: 10
        timeoutSeconds: 3
        successThreshold: 1
        failureThreshold: 3
      service:
        annotations: {}
      dagsterApiGrpcArgs: 
        - "--python-file"
        - "/opt/dagster/dagster_home/dg/repo.py"
        - "-p"
        - "4000"
      port: 4000

Hm, OK - and do you have the description of the job in 0.14.20 vs. 0.15.0 (with enabled:false)? I'd expect the YAML to be the same between the two, but it sounds like it is different

Hmmmmm

Gotcha - yeah the managed fields may be a red herring

My mistake, seems like some jobs still fail