https://dagster.io/ logo
#announcements
Title
# announcements
t

Travis Cline

05/28/2020, 8:21 PM
Jumping into a k8s-based deployment -- going to drop some initial thoughts into a thread here.
1. Initially expected to be able to control the run launcher configuration via the execution key in the pipeline config object.
2. not having more dynamic configuration (AFAICT so far) prevents some desirable patterns such as per-pipeline service account specification.
3. Is it true that the dagit UI can't launch pipeline runs that use run_launcher configuration?
a

alex

05/28/2020, 8:46 PM
dagit UI can’t launch pipeline runs
there should be a drop down to select between
start
and
launch
@nate for other notes
t

Travis Cline

05/28/2020, 8:47 PM
Ah yes, thanks @alex
a

alex

05/28/2020, 8:47 PM
we are working to remove that distinction in
0.8.0
it is confusing and goofy
n

nate

05/29/2020, 5:04 PM
hey Travis - on 1, definitely agree the split config between instance YAML and pipeline execution config is pretty rough right now, so agree here. for 2, are there other fields you’d like to be able to configure per-pipeline-run? or just service account?
cc @cat
t

Travis Cline

05/29/2020, 7:48 PM
definitely interested in per-run resource requests/limits (this would allow auto-scaling up when a pipeline run would use a GPU, for example)
Supplying the image on a per-pipeline is a likely requirement for many folks (would enable distinct dependency management between pipelines)
n

nate

06/05/2020, 3:01 PM
hey Travis, just following up here, I’ve updated our Celery/K8s configs: https://dagster.phacility.com/D3192 - so you should be able to configure these parameters on a per-pipeline basis now
t

Travis Cline

06/07/2020, 11:31 PM
badass!
n

nate

06/08/2020, 2:23 PM
let us know if you have other feedback, we’re actively working on improving this integration!