https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Charlie Bini

06/02/2022, 7:02 PM
is it possible to set a job's
executor_def
via a schedule? The job uses multiprocess as default, but there's one specific run that I'd like to run with the k8s_exectuor
j

johann

06/02/2022, 7:28 PM
You can set the k8s_job_executor in the executor def, but override it via run config to multiprocess by setting
Copy code
execution:
  config:
    multiprocess:
      ...
I’m curious what the use case is
c

Charlie Bini

06/02/2022, 8:08 PM
I have a dynamic graph that takes a config with a variable number API endpoints/queries. generally, each query takes about the same amount time to finish, but there's one endpoint that responds way slower...
so most queries will finish in ~5 min, but the one usually takes around an hour
grouping them all together on one large pod would result in an overprovisioned pod running for a long time
so using the k8s executor, I can break them up and the faster pods can terminate as soon as their done, leaving the long-running pod on its own using minimal resources
I have a few schedules with different configs based off of this job, and they all run for about the same duration, so there's not much need to use the k8s job executor
but this one is different
ultimately, I might use the k8s executor as the default, but still need to test it more thorougly