Hi I try to use GCS for intermediate storage in the K8s Cele

Question

Hi I try to use GCS for intermediate storage in the K8s Celery example It seams the <https docs dagster io deploying gcp> is not working on the K8s Helm deployment It works locally Appreciate any help hint

Noah K · Answer

Do you get a specific error

Noah K · Answer

Are you using GKE workload ID

Rolf Schick · Answer

I tried to mount the Service Account to each deployment via the google example here <https cloud google com kubernetes engine docs tutorials authenticating to cloud platform kubectl>

Rolf Schick · Answer

Unfortunately this was not passed on to the K8s job

Noah K · Answer

You wouldn t want to use that anyway it s been replaced by WorkloadID

Noah K · Answer

Check if that is enabled on your cluster

Noah K · Answer

This will be kind of awkward since the default helm chart doesn t allow setting a service account for the launched celery pods

Noah K · Answer

Noah K · Answer

Rolf Schick · Answer

OK so the preferred way is to link the helm created K8s Service Account to the GoogleService account right I guess then I ll have to link them right after the helm install

Noah K · Answer

Let s rewind which executor module are you using specifically

Noah K · Answer

Theres dagster celery dagster k8s and dagster k8s celery

Rolf Schick · Answer

Here is my Pipeline code

Noah K · Answer

Okay yeah

Noah K · Answer

So step 1 make sure workload ID is enabled

Noah K · Answer

Step 2 create a service account for your worker jobs that has the annotation for WorkloadID

Noah K · Answer

Step 3 set up a Google IAM service account and policy binding to allow access to the buckets you want

Noah K · Answer

Step 4 add ` configured service account name whatever ` on the executor definition in your pipeline code

Rolf Schick · Answer

Thanks a lot I ll come back once I have done all those steps Looking forward to get it finally running star struck

Noah K · Answer

It can be a bit annoying the first time but you get used to the setup I really need to wrap it into an operator at some point

Rolf Schick · Answer

Works now star struck Thanks very much