https://dagster.io/ logo
Powered by
#deployment-kubernetes
Title
# deployment-kubernetes
m

Matyas Tamas

03/03/2021, 5:11 AM
Hey - I noticed a strange issue with 
dagster-k8s/config
tags when trying to set some job spec values.
I have been using k8s pod_spec_config in tags to set nodeSelector to select a particular node pool on GKE. This appears to work when I set the key to snake_case, but fails when I set the key to camelCase.
This seems maybe expected given the way the tag config is passed to 
kubernetes.client.V1PodSpec(..., **user_defined_k8s_config.pod_spec_config)
in 
<http://dagster_k8s.jobs|dagster_k8s.jobs>
and given that 
V1PodSpec
takes snake case arguments: https://github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_pod_spec.py
But! Evidently I am missing something because: - the example in the documentation for passing user k8s configs through tags is camel case (mirroring the camel case of the k8s yaml keys) https://docs.dagster.io/deploying/kubernetes#custom-kubernetes-configuration - there is even an integration test that has more or less the same camel case syntax that is presumably passing without error. https://github.com/dagster-io/dagster/blob/master/integration_tests/test_suites/k8s-integration-test-suite/test_job_spec.py#L234
Here is an example of what I mean:
Copy code
@solid(
    tags={
        "dagster-k8s/config": {
            "pod_spec_config": {
                "node_selector": {"<http://cloud.google.com/gke-nodepool|cloud.google.com/gke-nodepool>": "cpu-workers"},
            }
        }
    }
)
def mysolid_snake_tags(_):
    return "done."

    # runs and returns correctly                                                                                                                                             



@solid(
    tags={
        "dagster-k8s/config": {
            "pod_spec_config": {
                "nodeSelector": {"<http://cloud.google.com/gke-nodepool|cloud.google.com/gke-nodepool>": "cpu-workers"},
            }
        }
    }
)
def mysolid_camel_tags(_):
    return "done."

    # ERROR:
    # dagster.core.errors.DagsterSubprocessError: During celery execution errors occurred in workers: [mysolid_camel_tags]: TypeError: __init__() got an unexpected keyword argument 'nodeSelector'
    #
    # Stack Trace:
    # File "/usr/local/lib/python3.8/site-packages/dagster_celery/core_execution_loop.py", line 79, in core_celery_execution_loop
    # step_events = result.get()
    # File "/usr/local/lib/python3.8/site-packages/celery/result.py", line 219, in get self.maybe_throw(callback=callback)
    # File "/usr/local/lib/python3.8/site-packages/celery/result.py", line 335, in maybe_throw self.throw(value, self._to_remote_traceback(tb))
    # File "/usr/local/lib/python3.8/site-packages/celery/result.py", line 328, in throw self.on_ready.throw(*args, **kwargs)
    # File "/usr/local/lib/python3.8/site-packages/vine/promises.py", line 234, in throw reraise(type(exc), exc, tb) File "/usr/local/lib/python3.8/site-packages/vine/utils.py", line 30, in reraise raise value
    #
    # File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/api.py", line 722, in pipeline_execution_iterator
    #     for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
    # File "/usr/local/lib/python3.8/site-packages/dagster_celery/core_execution_loop.py", line 155, in core_celery_execution_loop                                           
    #     raise DagsterSubprocessError(
@cat @sashank
s

sashank

03/03/2021, 5:19 PM
Hey @Matyas Tamas, taking a look at this now
m

Matyas Tamas

03/03/2021, 5:20 PM
Thanks!
s

sashank

03/03/2021, 6:14 PM
The reason for this is because we spread the config as **kwargs to the 
V1PodSpec
So the top level keys need to be 
snake_case
, but secondary level keys can be camel or snake case
We should probably convert the examples in the docs to exclusively use snake case for consistency
Here’s an example that demonstrates what’s going on:
Copy code
import kubernetes

# Directly passing the argument
job_container = kubernetes.client.V1Container(name="my_job", image="my_image:tag")

pod_spec = kubernetes.client.V1PodSpec(
    containers=[job_container],
    node_selector={"key": "value"},
)

# Using **kwargs
config_2 = {"node_selector": {"key": "value"}}

pod_spec_2 = kubernetes.client.V1PodSpec(
    containers=[job_container],
    **config_2,
)

assert pod_spec.to_dict() == pod_spec_2.to_dict()

# Using **kwargs with camelCase
exception_raised = False

config_3 = {"nodeSelector": {"key": "value"}}

try:
    pod_spec_3 = kubernetes.client.V1PodSpec(
        containers=[job_container],
        **config_3,
    )
except:
    exception_raised = True


assert exception_raised
In the example for the docs, 
nodeAffinity
is at the second level, so it’s not passed directly as a keyword arg and is rather just a dictionary key
Thanks for catching this, it’s definitely confusing. Tracking here to clarify docs: https://github.com/dagster-io/dagster/issues/3772
m

Matyas Tamas

03/03/2021, 6:35 PM
got it!
thanks for looking into that!
5 Views
Powered by