The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hello, I have a question about the helm chart, where can I define the resource limits and requests used by jobs

Hi <@U01RB6DCKCG>:
K8s allows you to set resource quotas and default resource requests for a namespace using ResourceQuotas and LimitRangers:
<https://kubernetes.io/docs/concepts/policy/resource-quotas/>

You can then override the defaults for each part of your dagster deployment. Resources for for dagit/daemon/userDeployments can be specified in the helm chart. Resources for jobs can be specified on tags for specific solids
<https://docs.dagster.io/deploying/kubernetes#custom-kubernetes-configuration>

If you’ll be using the same resource request tags across multiple solids, a recommended pattern is that you use a solid factory function that applies the tags

job that is executed still has default limit and requests of 1 cpu and 1 Gi memory

also tried to tag the pipeline with from dagster_k8s.job import K8S_RESOURCE_REQUIREMENTS_KEY

Are you using celery-k8s or just dagster-k8s?

runLauncher:
  # Type can be one of [K8sRunLauncher, CeleryK8sRunLauncher, CustomRunLauncher]
  type: K8sRunLauncher

Ah I was going to recommend tagging the pipeline, that didn’t work?

```@solid(
    tags={
        'dagster-k8s/config': {
            'container_config': {
                'resources': {
                        'requests': { 'cpu': '1', 'memory': '1Gi' },
                        'limits': { 'cpu': '2', 'memory': '8Gi' },
                }
            },
        },
    },
    required_resource_keys={'elasticsearch'},
    output_defs=[DynamicOutputDefinition()]
)```

That looks correct, `kubectl describe job` doesn’t show any of the added config?

And to be clear, you placed the same tag on the @pipeline?

pipeline is defined as
```@pipeline(
    mode_defs=[
        ModeDefinition(
            resource_defs={
                'elasticsearch': es_resource,
                'elasticsearch_async': es_async_resource,
                'io_manager': fs_io_manager,
                'loop': loop_resource,
                # 'gcs': gcs_resource,
            },
            executor_defs=default_executors + [celery_executor],
        )
    ]
)
def aggregation_pipeline():```

There’s an issue with the docs-
for the `K8sRunLauncher`, we only create one k8s job for the entire run, so the tag should be on the pipeline. Sorry about the confusion

For the `CeleryK8sRunLauncher`, we create a job per solid so you can vary resources at the solid level

Created a tracking issue to fix the docs :slightly_smiling_face: <https://github.com/dagster-io/dagster/issues/3824>