Hi team, we constantly see flakiness with one of our user-code-deployment repos that has lots of jobs & sensors with the 0.14.5 Dagit. For example, when viewing a sensor status, it flashes every few seconds between the correct page, blank screen, and an exception that “Exception: Location fabricator not in workspace” I am wondering if this is a problem with: a) Fabricator user-code-deployment not having enough pods/CPU to service the request from Dagit. We only have about 40% spike CPU usage on the pod. b) Dagit not having enough pods/CPU c) Some internal Dagit issue (maybe frequent auto-refresh?)

Hi Hebo - is the pod possibly restarting? Any logs from the pod you can share?

Thanks Daniel! No the pod doesn’t restart. I checked the logs but it is pretty much just our info logs. It might be related to location is doing too much sensor checks. I am wondering if there is a way to prevent the Dagit UI from frequent refreshing please? It seems to be able to load the data but then quickly auto-refresh within seconds

I'll double check, but I don't think the sensor page hits the server when it refreshes. I think that's more likely to be a symptom of the root problem (the server becoming inaccessible for some other reason) than the cause.

Yep, I think the root problem is definitely the location is inaccessible every few secs. However, if Dagit doens’t auto refresh frequently, we could at least still view the page. Now while I am on page, it flashes between working, blank, exception every few seconds.

We could address the refresh, but the server being down will cause other problems too. So I worry that that would bandaid over the root problem

The workspace tab is more stable but I also saw that the “Fabricator” location went completely missing.

Hmmmm, that's very unusual...

Glad that I asked. The replica count is currently 1 for Fabricator location. Let me just double CPU to see if it helps

I think this missing location thing is much more likely to be related to the root cause

right, we have 2 dagit replicas. We don’t define a workspace.yaml and I think Dagster auto-generates it?

Yeah, that's true. Is the fabricator location new?

I see..hmmm, the location has existed for a while. Let me see if I can isolate to the bad Dagit replica (Yeah, this is probably the root cause)

Very strange...

I see. Thanks for the help Daniel! Let me look into this. Also got this error now but I think it’s probably a syndrome not cause

dagster.core.errors.DagsterUserCodeUnreachableError: Could not reach user code server
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/context.py", line 555, in _load_location
    location = self._create_location_from_origin(origin)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/workspace/context.py", line 481, in _create_location_from_origin
    return origin.create_location()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/origin.py", line 306, in create_location
    return GrpcServerRepositoryLocation(self)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 557, in __init__
    self,
  File "/usr/local/lib/python3.7/site-packages/dagster/api/snapshot_repository.py", line 25, in sync_get_streaming_external_repositories_data_grpc
    repository_name,
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 260, in streaming_external_repository
    external_repository_origin
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 119, in _streaming_query
    raise DagsterUserCodeUnreachableError("Could not reach user code server") from e
The above exception was caused by the following exception:
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.UNKNOWN details = "Exception iterating responses: 'fabricator_mx_selection_net_new_merchants_base_instance_partition_set'" debug_error_string = "{"created":"@1651544410.089130103","description":"Error received from peer ipv4:172.30.227.74:3030","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Exception iterating responses: 'fabricator_mx_selection_net_new_merchants_base_instance_partition_set'","grpc_status":2}" >
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 117, in _streaming_query
    yield from response_stream
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in _next
    raise self

Where did you see that error?

In workspace locations. The error goes away few secs later.

That looks a bit more like what I would expect - an error trying to fetch a location that you think should be there, that then shows on the Workspace tab (that is more expected than the location being missing from the list altogether)

Hmmm..I restarted the dagit pods and bumped Fabricator CPU. Still seeing the problem. Maybe it IS something wrong with that “fabricator_mx_selection_net_new_stores_sales_prediction_base_partition_set” thing as I only see problems with that single job

You're still seeing one dagit replica with the location in the list on the Workspace tab, but another dagit replica with the location not in the list?

Hmmm..we are still seeing this problem. How do we know if it’s a dagit replica problem vs. Fabricator location sometimes isn’t responsive?

When you say "the problem" - we've discussed a couple of different problems here. Which problem exactly are you referring to?

1. It’s showing up but occasionally fails and then auto recover. (I think this is getting more stable now) 2. sensor page now flashes between working and loading every few seconds. (No longer seeing the exception)

and does 2) also manifest as the fabrictaor location totally missing from the Workspace tab? Or has that problem gone away?

Just stumbled upon this thread. @Hebo Yang Is this happening only for fabricator repo?

Thanks again Daniel for helping with this! It somehow became much much more stable now. We doubled the instance count, CPU, and mem for Dagit. We also doubled Fabricator CPU. Let me observe further to see what’s the behavior when the syndrome shows again

@daniel When I opened one of the sensors tick page from that repo, I see a

instigationStateOrError

request being made every second. Is this expected?

I'll check, that does seem like a lot (although that query shouldn't hit your user code server)

Also, at one point, I saw a 100s of requests being made for a sensor when I loaded the page. This also coincides with the loading issue Hebo mentioned

Was this the instigationStateOrError query? Or a different query?

Yeah, this might not be related to the user code server issue. But very much related to the sensor loading issue

Thanks, this is helpful context

Yes, I am able to reproduce it, but not consistently. Happens occasionally

When the sensor page is in the blank state, the whole sensor frame (including the counter is gone). I agree with Arun that it’s probably just making way too many requests on the backend to fetch all the states. Then sensor page will show up with partially loaded tick history, then go into blank state (probably trying to load the rest ticks) Most of the times when we go to the sensor page, we just need to enable the sensor or to check the most recent Skipped reason. If we could only fetch 10 or few historical ticks by default or somehow batch fetch, it would probably help. Also that most of our sensors have a frequency of 1m, so there are tons of tick histories. The recent ticks and tick history are usually not very useful for this kind of sensors.

How many sensors do you have in the repo?

Thanks Daniel! I think about 200+. Does the sensor view page (i.e. https://dagit.doordash.team/workspace/feature_jobs@fabricator/sensors/fabricator_features_sensor) fetch status for all sensors? I thought it would only fetch ticks for the particular one.

I'd expect it to just fetch from the one too - we'll see what we can do to make the performance here better, there's no reason we shouldn't be able to handle that many sensors

Yeah, I think that’s something we’d need to optimize. We are the MLPlatform team and we maintain the repo and service. Other teams add their ETL pipelines and we dynamically create sensors for them to check upstream dependency. Right now we create a sensor for each user pipeline. I think we could consolidate the sensors into few ones. That would also reduce the load on DB.