Noah K

12/23/2020, 2:40 PM
This was in the middle of a flurry of errors so I'm guessing it's some weird timing issue?


12/23/2020, 3:09 PM
Is it possible to post the other errors? This was the launcher that starts up runs in their own threads but all in the same process?

Noah K

12/23/2020, 3:10 PM
The other errors were legit pipeline failures, solid raised an exception.
It seems like about 50% failed with that error and 50% with the solid exception
This was with about 700 pipeline runs launched simultaneously
And yeah, each run in a thread, inside a celery worker process
Not something I've seen before hence the guess that's related to hundreds of runs failing almost immediately 😄
Might have been a connection overload on the gRPC daemon?
It doesn't have any logging output so hard to tell if it was sad from the rush of connections.