https://dagster.io/ logo
Join Slack
Powered by
How do you mount shared memory volumes within dags...
# deployment-kubernetes
i
How do you mount shared memory volumes within dagster k8s? I'm trying to run an op that needs GPUs and shared memory, I got the GPUs working but am having trouble with getting the shared memory volume to be mounted. Does anyone have any ideas on how to do this?
Copy code
@op(
    tags={
        "dagster-k8s/config": {
            "container_config": {
                "resources": {
                    "requests": {
                        "cpu": "10",
                        "memory": "80Gi",
                        "ephemeral-storage": "100Gi",
                        "<http://nvidia.com/gpu|nvidia.com/gpu>": "4",
                    },
                    "limits": {
                        "cpu": "10",
                        "memory": "80Gi",
                        "ephemeral-storage": "100Gi",
                        "<http://nvidia.com/gpu|nvidia.com/gpu>": "4",
                    },
                },
            },
            "pod_spec_config": {
                "node_selector": {"<http://k8s.amazonaws.com/accelerator|k8s.amazonaws.com/accelerator>": "nvidia-ampere-a10"},
                "volumes": [
                    {"name": "dshm", "emptyDir": {"medium": "Memory", "sizeLimit": "10Gi"}}
                ],
            },
        },
    },
)
a
Argh, I posted this on slack a while ago... it's probably gone in the slack-history-oblivion. From a first look your pod_spec_config.volumes section seems wrong. Why are you mounting an emptyvolume there?
or better... you have a volume section, but not a volumeMount section. I think that's what you are missing
daggy love 1
i
Yeah That makes sense, I added a volumeMount and things seem to be working (not getting errors for shared memor)
a
Can you post your new config here for reference?
i
yep, here's what I used
Copy code
@op(
    tags={
        "dagster-k8s/config": {
            "container_config": {
                "resources": {
                    "requests": {
                        "cpu": "10",
                        "memory": "80Gi",
                        "ephemeral-storage": "100Gi",
                        "<http://nvidia.com/gpu|nvidia.com/gpu>": "4",
                    },
                    "limits": {
                        "cpu": "10",
                        "memory": "80Gi",
                        "ephemeral-storage": "100Gi",
                        "<http://nvidia.com/gpu|nvidia.com/gpu>": "4",
                    },
                },
                "volumeMounts": [  # Mount the shared memory volume to the container
                    {
                        "name": "dshm",
                        "mountPath": "/dev/shm",  # Common mount path for shared memory
                    },
                ],
            },
            "pod_spec_config": {
                "node_selector": {"<http://k8s.amazonaws.com/accelerator|k8s.amazonaws.com/accelerator>": "nvidia-ampere-a10"},
                "volumes": [
                    {"name": "dshm", "emptyDir": {"medium": "Memory", "sizeLimit": "10Gi"}}
                ],
            },
        },
    },
)
a
@Tim Castillo It's worth moving this code fragment to a Dagster GH discusssion. Very useful when using GPUs
👀 1
12 Views
Open in Slack
PreviousNext
Powered by