Is there a way to scale Dagster across multiple servers Like

Question

Is there a way to scale Dagster across multiple servers Like flask behind haproxy

max · Answer

hi george which component of the system are you interested in scalin

George Pearse · Answer

The pipeline execution but I guess that s the container that the dagster pipelines container spins up when a pipeline starts running Guess Kubernetes or vertical scaling are the only ways to improve the performance of that part

max · Answer

execution is handled by the RunLauncher and Executor abstractions these are completely pluggable so you can pick from the available options in open source or write your own to their APIs

max · Answer

basically they let you control execution at two levels RunLaunchers govern where new runs are launched and Executors govern how the individual steps in a new run are launched

max · Answer

there are options that launch in Docker Kubernetes managed container runtimes like ECS as well as using OS processes and also options that let you use Celery for a queue worker model

max · Answer

which you pick will depend on exactly how you want to scale and what tradeoffs you want to make

max · Answer

<https docs dagster io deployment dagster instance run launcher> may be a good place to start

George Pearse · Answer

< max> I currently use DockerRunLauncher but would I get better performance if I just used DefaultRunLauncher Don t know if I get any benefits from that containerisation

max · Answer

ymmv there is some overhead associated with using containers for anything but the trade off is that you get isolation and repeatability