https://dagster.io/ logo
Powered by
d

David

03/12/2021, 9:04 AM
For dagster-celery-k8s I can't seem to get https://docs.dagster.io/deploying/kubernetes#custom-kubernetes-configuration working. Defined configuration in tags in the solid and pipeline decorators but they don't seem to be applied to the jobs. I am running the dagster helm chart in the new GKE autopilot Kubernetes cluster. But it seems unlikely to me that it has anything to do with that. The Kubernetes API is still the same and everything works as it should. It autoscales when new jobs are created just with the wrong resource limits set.
y

yuhan

03/12/2021, 6:18 PM
cc @rex
r

rex

03/12/2021, 6:24 PM
hey @David, could you paste your solid, and its tag config here? going to try to replicate your problem
d

David

03/12/2021, 7:36 PM
Copy code
@solid(
    tags={
        'dagster-k8s/config': {
            'container_config': {
                'resources': {
                        'requests': { 'cpu': '1', 'memory': '1Gi' },
                        'limits': { 'cpu': '2', 'memory': '8Gi' },
                }
            },
        },
    },
    required_resource_keys={'elasticsearch'},
    output_defs=[DynamicOutputDefinition()]
)
def lookup_all_records(context, organization_index: str):
    search = scroll_search(
        es=context.resources.elasticsearch,
        index=organization_index,
        query={
            'size': 10000,
            'query': {
                'match_all': {},
            },
        },
        scroll='1m',
    )

    count = 0

    for hits in search:
        yield DynamicOutput(hits, mapping_key=f'result_{count}')
        count += 1
Powered by