A second question - we lean on CUElang for all our config, and it would be good to use it in Dagster. We can manage resource connection strings, node names etc in production. Our pipeline for CUE is to compile the config down to a YAML file in CD and deploy that as a K8s secret. We’d do this for each repository. The secret is mounted into the pod using volumes like:

spec:
      volumes:
      - name: config
        secret:
          secretName: my-service-config
      containers:
        volumeMounts:
          - mountPath: "/etc/my-service-config"
            readOnly: true
            name: "config"

From what I can see on the docs we can import that into the container using

K8sRunLauncher

. Is that right? The main question is, can I merge configs? It would be good for all the production string values for resources to come from this precompiled YAML file, but op configuration we might want to handle in the launchpad - this would be great DX for our Data Scientists.

You can set config on the K8sRunLauncher to configure the spec of the k8s job/pod that it spins up, yeah - see volumeMounts and volumes here: https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L289-L300 That's different from the config that you set in the Launchpad: https://docs.dagster.io/concepts/ops-jobs-graphs/ops#op-configuration - the K8sRunLauncher config is k8s-specific and controls the spec of the k8s job/pod that is created, the Dagster op config that you set in the Launchpad is only checked once the pod is spun up and is not k8s-specific. They aren't merged together per se (since they operate in different parts of the stack), but you can absolutely use both.

Also, if you still have a need for merging configs, look over the functions in

dagster.utils.yaml_utils

; it has some helper functions in there that likely do what you’re looking for 🙂

I just discovered the default config options in the job here https://docs.dagster.io/concepts/ops-jobs-graphs/jobs-graphs#advanced-job-configuration 🤦🏽‍♂️ So the plan is to load in the YAML from the user code pod like how you said @daniel and then to merge it with some local default config with the functions @Alex Service. I’ll then just statically set the connection strings/passwords rather than provide as config so we don’t leak passwords.