https://dagster.io/ logo
Powered by
#deployment-kubernetes
Title
# deployment-kubernetes
s

Simon Späti

10/15/2020, 9:37 AM
Anyone tried to set kubernetes-port-setting for the 
dagster-run
-pod which gest scheduled when starting a pipeline? I'm trying to use spark which need a driver-host/port (as discussed with @bob and @johann). I managed to start a Spark session from the 
user-code
pod now (yay!! 🙂 ), but not yet from the scheduled pods. We managed to set the 
resources
(memory/cpu) successfully for a pipeline (and it's pod), but struggling with the port settings. (settings we used in the thread). Maybe @Kevin you did already such things? 🤔 Or is that setting not routed through to kubernetes yet? (if not, how would I achieve this)
This is what I tried to set on the pipeline tags. Only with the 
container_config
like this:
Copy code
{
    "container_config": {
        "resources": {
            "requests": {
                "cpu": "1",
                "memory": "1Gi"
            },
            "limits": {
                "cpu": "2",
                "memory": "2Gi"
            },
        },
        "ports": [
            {
                "name": " grpc",
                "containerPort": 3030,
                "protocol": "TCP"
            },
            {
                "name": "spark-driver",
                "containerPort": "4040",
                "protocol": "TCP"
            },
        ]
    }
}
or with the other key 
pod_spec_config
like this:
Copy code
{
  "container_config": {
    "resources": {
      "limits": {
        "cpu": "2",
        "memory": "2Gi"
      },
      "requests": {
        "cpu": "1",
        "memory": "1Gi"
      }
    }
  },
  "pod_spec_config": {
    "containers": {
      "ports": [
        {
          "name": " grpc",
          "containerPort": 3030,
          "protocol": "TCP"
        },
        {
          "name": " spark-driver",
          "containerPort": 4040,
          "protocol": "TCP"
        }
      ]
    }
  }
}
As said before, the resources block is correctly set, only the port part not.
On this line, I only see merge of 
resources
, tried to debug around there. Couldn’t print yet 
container_config
after merge, but will try again tomorrow. Otherwise I believe I can just add one more replace for 
ports
, at least that’s my take for now.
j

johann

10/15/2020, 8:03 PM
Hmm, @sashank is this the right way to specify the config?
s

Simon Späti

10/16/2020, 11:38 AM
Any idea on which pod and which time the get_user_defined_k8s_config(tags) is executed? I tried now to replace job.py, launcher.py from 
dagster_k8s
inside dagit & user-code pods (to add some debugging info), but I cannot see that anything is executed/printed.
j

johann

10/16/2020, 11:56 AM
The call site within dagster-k8s is just in the run-launcher, so that would execute within dagit
If you’d like we could hop on a zoom, I’m free for the next few hours and it might help us get to the bottom of this
s

sashank

10/16/2020, 2:44 PM
Yeah this is the right way to specify config
j

johann

10/16/2020, 2:45 PM
We would need to merge the two sources of labels
Wrong thread haha, mb
b

bob

10/20/2020, 7:59 PM
@Simon Späti It seems there’s some tricky parts to passing kubernetes-port-settings for the 
dagster-run
-pod, but you were on the right track. the code snippet below gives an example of using configs on the pipeline definition that will be passed down to the 
dagster-run
pod Note that: • Any k8s config (including container_config) needs to be under the 
dagster-k8s/config
tag key. You can take a look at 0.9.3 Release Notes for additional examples of using 
dagster-k8s/config
. • Unfortunately, k8s only allows “name” to contain alphanumeric (A-z0-9) characters, and complains when you try to use “spark-driver” :(((
Untitled.py
j

johann

10/20/2020, 9:20 PM
@Simon Späti would you be able to try this? I think we have an issue where you need to place it in the pipeline definition, as opposed to using Dagit
s

Simon Späti

10/21/2020, 2:52 PM
Thanks so much @bob and @johann for your effort, very much appreciated! I tried it and it works! The Pods get the right port opening 🎉 We're still checking the Spark set-up, still has a problem (same unresolved address), we're guessing that there is still a small issue with IP from the pod vs container. Will keep you posted. Two notes: • I think k8s allows dashes in the name? At leasst in my deployment for the user-code-pod, kubernetes sucessfully created with dashes (see attachment). • And OK noted johann, that's good to know. Because I did my tests interactively inside dagit. Didn't test now this in dagit, but for sure the exact tags from bob on pipelines and all solid did the trick (it's what we tried initially as well 🙂 )
6 Views
Powered by