https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
q

Quentin Gaborit

08/22/2023, 12:38 PM
Hi everyone, few questions about scalability of the web server and the daemon that I can't figure out without creating stress test scenarios. From what I understand, in k8s deployment the web server can be replicated but the daemon or user code server can not. • I suppose there is no concept of internal load balancing for these 2 services yet, and that the only option is vertical scaling when one starts to be saturated with too many Dagster events triggered jobs/sensors/schedules/partitions, is that correct? • Beyond optimising partition size, reducing schedule intervals, number of assets to materialise in general and setting minimum log-level, is there any set-up that seems being lighter for the web server, daemon and user code deployment services? • Do you have a range of CPU/MEM that should be allocated to the 3 services in a different deployments size (in terms of # job, # assets per job and schedule intervals? If not, any idea about a 1000 jobs of each 200 assets with as many schedules with 8 hours intervals? Thanks in advance for your help! 🙏
7 Views