https://dagster.io/ logo
Join Slack
Powered by
No rush on this, but getting errors from Dagit on ...
# ask-community
s
No rush on this, but getting errors from Dagit on Cloud when developing / testing backfills At one point this was due to a KeyError when generating partitioned config. Not sure what current issue is.
Copy code
Operation name: InstanceBackfillsQuery

Message: Internal Server Error (Trace ID: 5030567733442769130)

Path: ["partitionBackfillsOrError","results",10,"partitionSet"]

Locations: [{"line":22,"column":9},{"line":78,"column":3}]
🤖 1
Happening again, except with the further issue that backfill jobs can't be generated. Set them up to launch, and I get the GraphQL errors. If I try to generate the partititoned config from the Launchpad, I find a code error below -- money code is 
Exception calling application: Object of type datetime is not JSON serializable
, which I think is all pointing to poor handling of my flawed code.
Copy code
dagster.core.errors.DagsterUserCodeUnreachableError: dagster.core.errors.DagsterUserCodeUnreachableError: Could not reach user code server

Stack Trace:
File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 538, in _process_api_request
api_result = self._handle_api_request(request, instance, user_code_launcher)
File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 338, in _handle_api_request
serialized_partition_config_or_error = client.external_partition_config(
File "/dagster/dagster/grpc/client.py", line 181, in external_partition_config
res = self._query(
File "/dagster/dagster/grpc/client.py", line 110, in _query
raise DagsterUserCodeUnreachableError("Could not reach user code server") from e

The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Exception calling application: Object of type datetime is not JSON serializable"
debug_error_string = "{"created":"@1652500120.293077518","description":"Error received from peer ipv4:10.100.81.96:4000","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Exception calling application: Object of type datetime is not JSON serializable","grpc_status":2}"
>

Stack Trace:
File "/dagster/dagster/grpc/client.py", line 107, in _query
response = getattr(stub, method)(request_type(**kwargs), timeout=timeout)
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)


  File "/dagster-graphql/dagster_graphql/implementation/utils.py", line 48, in _fn
    return fn(*args, **kwargs)
  File "/dagster-graphql/dagster_graphql/implementation/fetch_partition_sets.py", line 92, in get_partition_config
    result = graphene_info.context.get_external_partition_config(
  File "/dagster/dagster/core/workspace/context.py", line 221, in get_external_partition_config
    return self.get_repository_location(
  File "/ursula/ursula/user_code/workspace.py", line 358, in get_external_partition_config
    result = self.api_call(
  File "/ursula/ursula/user_code/workspace.py", line 238, in api_call
    return dagster_cloud_api_call(
  File "/ursula/ursula/user_code/workspace.py", line 69, in dagster_cloud_api_call
    for result in gen_dagster_cloud_api_call(
  File "/ursula/ursula/user_code/workspace.py", line 164, in gen_dagster_cloud_api_call
    raise DagsterUserCodeUnreachableError(error_infos[0].to_string())
r
Hey Stephen - thanks for the report. We’re looking into this. After fixing the serialization error in your code, are you still encountering this error?
this was a regression on our part. we can push a fix out on monday
🎉 1
d
Besides the Dagit error which we'll fix on our side like Rex mentioned, I think we've cleaned up that error handling path a bit in later versions of the agent - no rush at all, but if you update your code image to the latest version of dagster/dagster-cloud, I bet that "Object of type datetime is not JSON serializable" error will have a nice stack trace within your code and not be wrapped up in that cryptic DagsterUserCodeUnreachableError message
👌 1
s
yeah, still getting the error. thanks team!
r
Just sent out the fix, let me know if still have trouble loading your backfills page from the status menu
❤️ 1
s
looking good now - thanks rex
12 Views
Open in Slack
PreviousNext
Powered by