https://dagster.io/ logo
#ask-community
Title
# ask-community
g

George Pearse

04/05/2022, 9:58 AM
Facing a really weird bug, everything seems to be working fine, but the scheduler just isn't triggering any jobs. Trying to check the daemon logs, just a bit of faff,
c

claire

04/05/2022, 4:49 PM
Hey George. Were you able to check the daemon logs? If so, mind pasting the output here?
p

prha

04/05/2022, 11:18 PM
Hey George… We’ve seen issues like this, especially when
dagit
and
dagster-daemon
processes are out of sync (potentially run in different python environments). Do you see a
Unloadable schedules
section when you navigate to
Status > Schedules
in dagit?
We do have a fix for this whole class of issues in this week’s upcoming release.
❤️ 1
g

George Pearse

04/06/2022, 7:10 AM
Copy code
Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 107, in _query
    response = getattr(stub, method)(request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
2022-04-05 10:49:04 +0000 - dagster.daemon.SchedulerDaemon - WARNING - Schedule radiologist_report_schedule was started from a location k8s-prod-user-code-1 that can no longer be found in the workspace, or has metadata that has changed since the schedule was started. You can turn off this schedule in the Dagit UI from the Status tab.
2022-04-05 10:49:04 +0000 - dagster.daemon.SchedulerDaemon - WARNING - Schedule hospital_model_reports_schedule was started from a location k8s-prod-user-code-1 that can no longer be found in the workspace, or has metadata that has changed since the schedule was started. You can turn off this schedule in the Dagit UI from the Status tab.
2022-04-05 10:49:04 +0000 - dagster.daemon.SchedulerDaemon - WARNING - Schedule identifier_mapping_schedule was started from a location k8s-prod-user-code-1 that can no longer be found in the workspace, or has metadata that has changed since the schedule was started. You can turn off this schedule in the Dagit UI from the Status tab.
2022-04-05 10:49:04 +0000 - dagster.daemon.SchedulerDaemon - WARNING - Schedule ml_cxr_datalake_schedule was started from a location k8s-prod-user-code-1 that can no longer be found in the workspace, or has metadata that has changed since the schedule was started. You can turn off this schedule in the Dagit UI from the Status tab.
2022-04-05 10:49:04 +0000 - dagster.daemon.SchedulerDaemon - WARNING - Schedule series_metadata_schedule was started from a location k8s-prod-user-code-1 that can no longer be found in the workspace, or has metadata that has changed since the schedule was started. You can turn off this schedule in the Dagit UI from the Status tab.
2022-04-05 10:49:04 +0000 - dagster.daemon.SchedulerDaemon - WARNING - Schedule study_metadata_schedule was started from a location k8s-prod-user-code-1 that can no longer be found in the workspace, or has metadata that has changed since the schedule was started. You can turn off this schedule in the Dagit UI from the Status tab.
2022-04-05 10:50:04 +0000 - dagster.daemon.SensorDaemon - WARNING - Could not load location docker_dev_dagster_pipelines to check for sensors due to the following error: dagster.core.errors.DagsterUserCodeUnreachableError: Could not reach user code server
Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 107, in _load_location
    location = self._create_location_from_origin(origin)
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/workspace.py", line 126, in _create_location_from_origin
    return origin.create_location()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/origin.py", line 306, in create_location
    return GrpcServerRepositoryLocation(self)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 526, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File "/usr/local/lib/python3.7/site-packages/dagster/api/list_repositories.py", line 19, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 164, in list_repositories
    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 110, in _query
    raise DagsterUserCodeUnreachableError("Could not reach user code server") from e
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "DNS resolution failed for docker_dev_dagster_pipelines:4000: C-ares status is not ARES_SUCCESS qtype=A name=docker_dev_dagster_pipelines is_balancer=0: Domain name not found"
    debug_error_string = "{"created":"@1649155804.332119663","description":"DNS resolution failed for docker_dev_dagster_pipelines:4000: C-ares status is not ARES_SUCCESS qtype=A name=docker_dev_dagster_pipelines is_balancer=0: Domain name not found","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}"
>
Presumably it's something like the name in the workspace.yaml not matching how it's defined in kubernetes, I just don't get how it's working for dagit and not the daemon
Resolved the bug but not seeing the executions on the status page (they are definitely running succesfully)
28 Views