https://dagster.io/ logo
#ask-community
Title
# ask-community
k

Korntewin Boonchuay

04/14/2023, 6:32 AM
Hello, I’ve a question regarding dagit bottleneck on OSS deployment. Are there any possibilities that launching many executors (about 400-500 executors) will cause performance bottleneck because of log streaming from executor back to dagit?
d

daniel

04/14/2023, 3:02 PM
Hi - I wouldn't expect dagit to be a bottleneck here. The way this works is: • ops write to the dagster database • dagit (when you happen to have a run loaded) streams data from the database for the particular run that you're looking at So if anything is going to be the bottleneck, it will be the database handling all the writes from the 300-400 ops at once - but there are lots of quite scalable cloud database solutions out their like RDS, so I would be surprised if that was a bottleneck as long as you have a beefy enough database to handle that many connections
k

Korntewin Boonchuay

04/14/2023, 5:35 PM
Thanks a lot! Then it might be because of our database 😱. The problem we faced is 1. Running one or two jobs on separate fargate, the job will be quite fast (1-2 minute per job) 2. Running 400-500 on separate fargate, the job will be super slow (10-15 minutes per job) 3. from logging, the bottleneck seems to be something unobservable during the beginning and the end of the op’s (not the logic inside ops)
d

daniel

04/14/2023, 5:35 PM
What kind of database are you using?
k

Korntewin Boonchuay

04/14/2023, 5:36 PM
RDS with two cpu cores
d

daniel

04/14/2023, 5:37 PM
I could imagine that needing some more CPUs for that many ops running at once, yeah
RDS has some pretty good performacne monitoring tools I think - you could check if its getting capped out on CPU
k

Korntewin Boonchuay

04/14/2023, 5:38 PM
ohh you’re right! Thanks!
Yeah, definitely RDS problem! The RRS hit 100% constantly during peak load 😂. Thanks again!
condagster 1
🎉 1