Guillaume Onfroy
07/14/2023, 2:59 PMcode location
pod and not in a separate pod (e.g. from the k8sRunLauncher
)? I was hoping to run steps directly within the initial user deployment container to avoid really long cold starts we get with separate jobs.
It feels like the runLauncher
config should allow something like that:
runLauncher:
type: DefaultRunLauncher
But it doesn't...rex
07/14/2023, 3:16 PMtype: CustomRunLauncher
.
runLauncher:
type: CustomRunLauncher
customRunLauncher:
module: dagster
class: DefaultRunLauncher
config: {}
See this previous discussion.Guillaume Onfroy
07/14/2023, 3:17 PMGuillaume Onfroy
07/14/2023, 3:46 PMGuillaume Onfroy
07/14/2023, 3:47 PMGuillaume Onfroy
07/14/2023, 3:47 PMrex
07/14/2023, 3:50 PMDefaultRunLauncher
will run the job directly in your long standing code server. But if you are running multiple jobs in this code server, execution will get bogged down since your code server may not have enough resources to handle this simultaneous execution.
I would start by beefing up your code server resources.Guillaume Onfroy
07/14/2023, 4:03 PMGuillaume Onfroy
07/14/2023, 4:03 PMK8sRunLauncher
RUN_ENQUEUED >> 133 sec >> RUN_START >> 107 sec >> STEP_START
DefaultRunLauncher
RUN_ENQUEUED >> 103 sec >> RUN_START >> 98 sec >> STEP_START
Guillaume Onfroy
07/14/2023, 4:05 PM