19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
dagster-de
dagster-cube
gigs-freelance
dagster-wandb
dagster-machine-learning
dagster-bigquery
tools
dagster-dask
dagster-snowflake
dagster-releases
data-platform-design
events
dagster-serverless
dagster-noteable
introductions
dagster-airflow
dagster-cloud
random
dagster-dbt
faq-read-me-before-posting
dagster-airbyte
dagster-spatial
dagster-support
announcements
github-discussions
豆瓣酱帮
dagster-feedback
dagstereo
dagster-ecs
dagster-kubernetes
jobs
dagster-showcase
Title
f
Francis Niu
09/27/2021, 2:30 AMHi Team,
I deployed in k8s with helm and separated dagster and user code. I have 2 questions:
1. Since user code container is a long running service, why dagster need to launch a new container with same user code image for each execution?
2. I need to mount a config file into user code container for different environment (qa/uat/prod). How can I mount the same config into the new launched job container?
d
daniel
09/27/2021, 2:41 AMhi Francis -
1. It doesn't actually have to do this if you really don't want it to (the DefaultRunLauncher launches runs within the user code container, and its possible to configure the helm chart so that it uses the DefaultRunLauncher instead of the K8sRunLauncher using the 'customRunLauncher' field). However there are a lot of advantages to putting each run in its own job/container which is why its the default -a couple of the big ones are that it gives you a lot more isolation between runs, it makes it easier to take advantage of k8s autoscaling features, and it allows you to update your pipeline code without disrupting runs that are in the middle of executing.
2. As luck would have it, we're planning to add the ability to specify volume mounts in the K8sRunLauncher with the upcoming release later this week. Once that lands you should be able to configure your job containers to have these config files (either by configuring the run launcher if you want it to apply to all runs, or as tags on individual pipelines if you only want it for some pipelines)
❤️ 1
f
Francis Niu
09/27/2021, 2:56 AMA dagster instance can orchestrate multiple user code deployments (
workspace.serversd
daniel
09/27/2021, 3:09 AMRight now there are two options - at the instance level by configuring the run launcher (which applies to all user code deployments) or at the pipeline level via tags - unfortunately there isn’t a way to apply config to all runs launched in a given user code deployment, but it’s an entirely reasonable feature request
f
Francis Niu
09/27/2021, 3:32 AMI see. I understood the advantages of K8sRunLauncher. But applying same configurations to individual user code deployments and corresponding executor seems to be a complex design. Perhaps, one shared common configuration in one dagster instance for all user code deployments would be a simple practice.
For now I plan to use DefaultRunLauncher. I still use
<http://docker.io/dagster/dagster-celery-k8s|docker.io/dagster/dagster-celery-k8s>runLauncher:
# Type can be one of [K8sRunLauncher, CeleryK8sRunLauncher, CustomRunLauncher]
type: CustomRunLauncher
config:
customRunLauncher:
module: dagster.core.launcher.default_run_launcher
class: DefaultRunLauncher
config: {}d
daniel
09/27/2021, 3:45 AMI’d expect that to work, yeah
f
Francis Niu
09/27/2021, 4:09 AMThank you very much! It saved my day.
:condagster: 1