Hi all :wave:, I'm very new to Dagster and I'm wor...
# deployment-kubernetes
Hi all 👋, I'm very new to Dagster and I'm working on deploying dagster along with a user code deployment to a local k8s cluster (using Helm). I'm using
along with the
with a very simple 2 step pipeline. I'm able to get my pipeline to start, but I'm using GCS as my io_manager. Once my first step starts, I see this error message:
Copy code
Initialization of resources [gcs, io_manager] failed.
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see <https://cloud.google.com/docs/authentication/getting-started>
My question is, what's the proper way to set this env variable? My thought is to take the following steps: 1. create a k8s secret that has the contents of my private key file. 2. define a k8s volume with the secret 3. mount the secret volume to the
job's container 4. set
to point to key file in volume mount I'm not exactly sure how to accomplish step 3 though. Does anyone know if this is the proper way to use GCS for intermediate storage? Is there a better way? Thanks in advance!
Hi @Ginger Balmat, unfortunately the volume_mounts config for k8s currently isn’t exposed in the k8s_job_executor. It’s a small change- could have it out in the release this thursday.
It’s worth mentioning that if this is on GKE, the default service account could be given access to your GCP bucket. The above will be better though, for minimum access
Thanks so much for the clarification @johann! I think what I'll do is use Minio locally and then rely on the service account for my GKE deployment.