Hey everyone. I have a QueuedRunCoordinator that limits how many runs I can have of a specific op (which effectively limits the number of jobs) at one time. This is because I have a sensor that kicks off hundreds (possibly thousands in the future) jobs at once. I’m currently running into an error where each job in the queue fails due to a “DagsterRepositoryLocationLoadError”, which itself is caused by an “_InactiveRpcError”. Any ideas how I could fix?

@johann any ideas on this?

I may be wrong. But it seems to me that you specify

grpc_server

in

workspace.yaml

to load a repository. So,

grpc

server might not start as it was expected. Therefore, grpc client can not connect and load a user repository code.

_InactiveRpcError

means the grpc server for your repository location didn’t respond

Hmm ok. What's confusing me is that the grpc server does start and is used by the jobs actually running (not in the queue)

What deployment are you using?

what do you mean? at the moment I’m working on a single, self hosted machine.

And using the DefaultRunLauncher?

yes

is used by the jobs actually running (not in the queue)

does the rpc error not happen every time? How did those jobs manage to start running?

It does happen every time. I think it occurs when around 200 jobs are in the queue?

One possibility is that with everything on one box, the grpc server is getting starved and doesn’t respond to new requests. This happens once you have a lot of runs in progress?

yeah, 20-40

You could try setting a lower limit on the number of runs and see if that helps the error

ok! a bit confused because the error didn't occur before, but will definitely try that and get back to you!

Long term if you want more parallelism you’ll want to look in to other options for run launchers that kick off new containers

If the number of jobs was an issue, you lower the # of jobs or run a grpc server in a seperate container.

yeah I have a k8s setup possible but I was using this for the short term.