19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#dagster-kubernetes
Title
# dagster-kubernetes
d
Dusty Shapiro
02/08/2023, 2:36 PMIs there a way to get completed run pods to get cleaned up/removed automatically? Thanks again!
a
Adam Bloom
02/08/2023, 2:41 PM1. Set the job ttl https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment
2. Look to airbyte for inspiration and deploy a modified version of its pod-sweeper script
We’re currently doing 2, but I’ll probably switch to 1 at some point.
d
Dusty Shapiro
02/08/2023, 3:02 PMGood stuff, thanks. Looks like I would customize this in the Helm chart at
runLauncher.config.k8sRunLauncher.resourcesd
daniel
02/08/2023, 3:03 PMthere's some more specific guidance later on in that page: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#kubernetes-job-and-pod-ttl-management
The example shows setting it on particular jobs but you could also use
Copy code
runLauncher.config.k8sRunLauncher.runK8sConfig.jobSpecConfiga
Adam Bloom
02/08/2023, 3:03 PMWhoops, meant to paste the anchor link. Thanks Daniel!
blob highfive 1
d
Dusty Shapiro
02/08/2023, 3:09 PMGood stuff, thanks again Adam and Daniel!
g
Grigorii Kushnir
02/08/2023, 3:15 PMI was also searching for that today.
Please, tell me how should I configure the ttl if I have a config like this for the code location pod:
Copy code
...
- name: DAGSTER_CLI_API_GRPC_CONTAINER_CONTEXT
value: |
{
"k8s": {
"service_account_name": "dorche-dagster",
"namespace": "dorche-dagster-ns",
"env_secrets": [
"aws",
"postgres",
"snowflake"
],
"env_vars": [
"ENVIRONMENT=${ENVIRONMENT}",
"GIT_HASH=${GIT_HASH}",
"DAGSTER_HOME=/home/dagster",
"SNOWFLAKE_KEY_PATH=/app/secret/snowflake_key.txt"
],
...I couldn't find an answer and I hope I can hit the default ttl of 1 day, if that applies here.
@daniel can I specify this ttl somehow in the run_launcher dict in the dagster.yaml?
I imagine the answer is somewhere here but I can't make use of it:
d
daniel
02/08/2023, 3:23 PMYeah it's in job_spec_config
run_k8s_config:
job_spec_config:
ttl_seconds_after_finished: 7200
g
Grigorii Kushnir
02/08/2023, 3:25 PM Copy code
env:
- name: DAGSTER_CLI_API_GRPC_CONTAINER_CONTEXT
value: |
{
"k8s": {
"run_k8s_config": {
"job_spec_config":
{
"ttl_seconds_after_finished": 7200
}
}Will something like this work in the code location pod config?
d
daniel
02/08/2023, 3:25 PMThat would also work, yeah
g
Grigorii Kushnir
02/08/2023, 3:26 PMThank you!
Is the default 1 day?
d
daniel
02/08/2023, 3:26 PMThat will depend on your cluster I believe - dagster doesn't control the default
dagster angel 1
g
Grigorii Kushnir
02/08/2023, 3:41 PMTested, works!
9 Views