https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
t

takan

07/26/2021, 5:50 AM
Hi all, I’m testing my GKE deployment and am wondering how to properly set up with GCS. Which part in the tutorial (https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm), should I change so that I use my GCS buckets instead of S3?
j

johann

07/26/2021, 12:57 PM
Hi @takan - per the thread above https://dagster.slack.com/archives/C014N0PK37E/p1627304164218100?thread_ts=1627072705.216700&cid=C014N0PK37E it’s unfortunately not very well supported right now, but can be done by changing the default service account
t

takan

07/26/2021, 1:55 PM
@johann Thanks a lot for the response! Okay, I changed with my default service account for now but exactly which part should I change to use GCS? I’m playing around but ended up having
Unable to locate credentials
errors still.
Copy code
# Launch Execution
resources:
  io_manager:
    config:
      s3_bucket: "test-bucket" # <- what should I put here? I tried gcs_bucket but unexpected config error...
solids:
  multiply_the_word:
    config:
      factor: 0
    inputs:
      word: ""
Copy code
# values.yaml <- not exactly sure which part I should change for GCS :( 
dagster-user-deployments:
  enabled: true
  deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "<http://docker.io/dagster/user-code-example|docker.io/dagster/user-code-example>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/example_project/example_repo/repo.py"
      port: 3030
      envSecrets:
        - name: dagster-aws-access-key-id
        - name: dagster-aws-secret-access-key

runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      envSecrets:
        - name: dagster-aws-access-key-id
        - name: dagster-aws-secret-access-key
j

johann

07/26/2021, 2:01 PM
The s3 io manager is defined in the mode of the pipeline code. For the example image, that’s https://github.com/dagster-io/dagster/blob/master/examples/deploy_k8s/example_project/example_repo/repo.py#L32 To switch to gcs, you’ll want to build your own user code image
t

takan

07/26/2021, 2:20 PM
Ah okay I need to define there in the pipeline, got it 🙂 I’ll try that. Thanks on this one! @johann
Hi @johann, so I changed the pipeline file (repo.py and corresponding run_config/*.yamls) to use gcp for io manager and I built my own image and now would like to use it in dagit. I’m following the Separately Deploying Dagster infrastructure and User Code and deployed dagster successfully. Now, I’m trying to deploy the user-code with my changed values.yaml (with
$ helm upgrade --install user-code dagster/dagster-user-deployments -f user-dep-values.yaml
) looking like below.
Copy code
dagster-user-deployments:
  enabled: true
  deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "<http://gcr.io/my-own-image|gcr.io/my-own-image>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/example_project/example_repo/repo.py"
      port: 3030
But dagit still shows the example user-code (“docker.io/dagster/user-code-example”), not my own image 😕 Am I missing something here?
j

johann

07/28/2021, 1:38 PM
I would • verify that your k8s pods are using the new image • click the refresh on the repository locations in dagit • verify that you’re seeing dagit actually display the example image, not just that your image has the same pipelines or something like that • what image are you using for dagit? The default in the helm chart?
👍 1