Hi all, I'm facing the issue of slowness like I ha...
# random
q
Hi all, I'm facing the issue of slowness like I have a job which takes 3 to 4 minutes to get complete when I run a single job it completes in 3 to 4 minutes but when I run around 32 concurrent jobs the time for each job reach to 1 hour or more even though my ec2 server is large and using one 20% of its resources only, i'm not sure why it's happening that by increasing the concurrent jobs speed getting slow.
Anyone?
s
Hi @Qumber Ali - I can't think of a straightforward way that Dagster would be causing this slowdown. Are you able to tell where the bottleneck is? I.e. what part of the jobs are running slow?
q
@johann Can you please have a look at this too?
j
It looks like Daniel responded to your thread in https://dagster.slack.com/archives/C01U954MEER/p1636957546191700