19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
is it possible to set a job's `executor_def` via a...
# ask-communityc
Charlie Bini
06/02/2022, 7:02 PMis it possible to set a job's
executor_defj
johann
06/02/2022, 7:28 PMYou can set the k8s_job_executor in the executor def, but override it via run config to multiprocess by setting
Copy code
execution:
config:
multiprocess:
...johann
06/02/2022, 7:30 PMAnd you can return custom run config with https://docs.dagster.io/concepts/partitions-schedules-sensors/schedules#a-schedule-that-provides-custom-run-config-and-tags
johann
06/02/2022, 7:30 PMI’m curious what the use case is
c
Charlie Bini
06/02/2022, 8:08 PMI have a dynamic graph that takes a config with a variable number API endpoints/queries. generally, each query takes about the same amount time to finish, but there's one endpoint that responds way slower...
Charlie Bini
06/02/2022, 8:08 PMso most queries will finish in ~5 min, but the one usually takes around an hour
Charlie Bini
06/02/2022, 8:09 PMgrouping them all together on one large pod would result in an overprovisioned pod running for a long time
Charlie Bini
06/02/2022, 8:11 PMso using the k8s executor, I can break them up and the faster pods can terminate as soon as their done, leaving the long-running pod on its own using minimal resources
Charlie Bini
06/02/2022, 8:13 PMI have a few schedules with different configs based off of this job, and they all run for about the same duration, so there's not much need to use the k8s job executor
Charlie Bini
06/02/2022, 8:13 PMbut this one is different
Charlie Bini
06/02/2022, 8:14 PMultimately, I might use the k8s executor as the default, but still need to test it more thorougly