I'm facing the issue of slowness like I have a job which takes 3 to 4 minutes to get complete when I run a single job it completes in 3 to 4 minutes but when I run around 32 concurrent jobs the time for each job reach to 1 hour or more even though my ec2 server is large and using one 20% of its resources only, i'm not sure why it's happening that by increasing the concurrent jobs speed getting slow.
11/15/2021, 5:08 PM
Hi @Qumber Ali - I can't think of a straightforward way that Dagster would be causing this slowdown. Are you able to tell where the bottleneck is? I.e. what part of the jobs are running slow?