Hey, I am trying to specify the pod_spec as a tag when running my pipeline with the following config:

{
  "pod_spec_config": {
    "volumes": [
      {
        "name": "task-pv-storage",
        "persistentVolumeClaim": {
          "claimName": "task-pv-claim"
        }
      }
    ],
    "containers": {
      "volumeMounts": [
        {
          "mountPath": "/app/data",
          "name": "task-pv-storage"
        }
      ]
    }
  }
}

The idea is to have K8s Jobs use data from a persistent volume instead of having the solid download from GCloud Storage. However, when I run my pipeline with this tag the following error pops out. I've been reading through the code and my guess is that the problem is in this line https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-k8s/dagster_k8s/job.py#L485. Since volumes are already specified as kwarg, this conflicts with my volumes spec in the config. What should I do here? I read the other day here in slack that somebody said that volumes could not be specified for K8s jobs at the moment, but I want to be sure that it's the same issue I'm having before proceeding to store my data in GCloud Storage and stop attempting to use PVs.

Yeah, this looks like an issue on our end - we currently hard code a volume for our

dagster.yaml

configuration file. Instead of overriding the field, we should just add the user’s specified volumes to Dagster’s required volumes.

Created issue at: https://github.com/dagster-io/dagster/issues/3871

Cool, thanks!