Anyone tried to set kubernetes-port-setting for th...
# deployment-kubernetes
s
Anyone tried to set kubernetes-port-setting for the
dagster-run
-pod which gest scheduled when starting a pipeline? I'm trying to use spark which need a driver-host/port (as discussed with @bob and @johann). I managed to start a Spark session from the
user-code
pod now (yay!! 🙂 ), but not yet from the scheduled pods. We managed to set the
resources
(memory/cpu) successfully for a pipeline (and it's pod), but struggling with the port settings. (settings we used in the thread). Maybe @Kevin you did already such things? 🤔 Or is that setting not routed through to kubernetes yet? (if not, how would I achieve this)
This is what I tried to set on the pipeline tags. Only with the
container_config
like this:
Copy code
{
    "container_config": {
        "resources": {
            "requests": {
                "cpu": "1",
                "memory": "1Gi"
            },
            "limits": {
                "cpu": "2",
                "memory": "2Gi"
            },
        },
        "ports": [
            {
                "name": " grpc",
                "containerPort": 3030,
                "protocol": "TCP"
            },
            {
                "name": "spark-driver",
                "containerPort": "4040",
                "protocol": "TCP"
            },
        ]
    }
}
or with the other key
pod_spec_config
like this:
Copy code
{
  "container_config": {
    "resources": {
      "limits": {
        "cpu": "2",
        "memory": "2Gi"
      },
      "requests": {
        "cpu": "1",
        "memory": "1Gi"
      }
    }
  },
  "pod_spec_config": {
    "containers": {
      "ports": [
        {
          "name": " grpc",
          "containerPort": 3030,
          "protocol": "TCP"
        },
        {
          "name": " spark-driver",
          "containerPort": 4040,
          "protocol": "TCP"
        }
      ]
    }
  }
}
As said before, the resources block is correctly set, only the port part not.
On this line, I only see merge of
resources
, tried to debug around there. Couldn’t print yet
container_config
after merge, but will try again tomorrow. Otherwise I believe I can just add one more replace for
ports
, at least that’s my take for now.
j
Hmm, @sashank is this the right way to specify the config?
s
Any idea on which pod and which time the get_user_defined_k8s_config(tags) is executed? I tried now to replace job.py, launcher.py from
dagster_k8s
inside dagit & user-code pods (to add some debugging info), but I cannot see that anything is executed/printed.
j
The call site within dagster-k8s is just in the run-launcher, so that would execute within dagit
If you’d like we could hop on a zoom, I’m free for the next few hours and it might help us get to the bottom of this
s
Yeah this is the right way to specify config
j
We would need to merge the two sources of labels
Wrong thread haha, mb
b
@Simon Späti It seems there’s some tricky parts to passing kubernetes-port-settings for the
dagster-run
-pod, but you were on the right track. the code snippet below gives an example of using configs on the pipeline definition that will be passed down to the
dagster-run
pod Note that: • Any k8s config (including container_config) needs to be under the
dagster-k8s/config
tag key. You can take a look at 0.9.3 Release Notes for additional examples of using
dagster-k8s/config
. • Unfortunately, k8s only allows “name” to contain alphanumeric (A-z0-9) characters, and complains when you try to use “spark-driver” :(((
j
@Simon Späti would you be able to try this? I think we have an issue where you need to place it in the pipeline definition, as opposed to using Dagit
s
Thanks so much @bob and @johann for your effort, very much appreciated! I tried it and it works! The Pods get the right port opening 🎉 We're still checking the Spark set-up, still has a problem (same unresolved address), we're guessing that there is still a small issue with IP from the pod vs container. Will keep you posted. Two notes: • I think k8s allows dashes in the name? At leasst in my deployment for the user-code-pod, kubernetes sucessfully created with dashes (see attachment). • And OK noted johann, that's good to know. Because I did my tests interactively inside dagit. Didn't test now this in dagit, but for sure the exact tags from bob on pipelines and all solid did the trick (it's what we tried initially as well 🙂 )