https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
d

Dusty Shapiro

09/28/2022, 4:06 PM
Attempting to deploy Dagster via a Helm chart to K8s, but receiving this error/stacktrace:
Copy code
WARNING - Could not load location data-team-dag-code to check for schedules due to the following error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/workspace.py", line 131, in _load_location
    location = self._create_location_from_origin(origin)
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/workspace.py", line 150, in _create_location_from_origin
    return origin.create_location()
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/origin.py", line 333, in create_location
    return GrpcServerRepositoryLocation(self)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/repository_location.py", line 561, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File "/usr/local/lib/python3.7/site-packages/dagster/_api/list_repositories.py", line 19, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 169, in list_repositories
    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 115, in _query
    raise DagsterUserCodeUnreachableError("Could not reach user code server") from e

The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1664381076.964556226","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3260,"referenced_errors":[{"created":"@1664381076.964555536","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
>

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 112, in _query
    response = getattr(stub, method)(request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
d

daniel

09/28/2022, 4:12 PM
Hey Dusty - are there any clues in the logs for the user code deployment pod? If it failed to start up there would hopefully be an error there
one possible gotcha... you might need to include dagster-k8s in the dependencies as well
d

Dusty Shapiro

09/28/2022, 4:13 PM
Yea, let me log out that pod
d

daniel

09/28/2022, 4:13 PM
If it's failing with an error like "could not import dagster_k8s..." that'd be why
d

Dusty Shapiro

09/28/2022, 4:13 PM
and add that to the Dockerfile. Also hello again Daniel and thanks
d

daniel

09/28/2022, 4:13 PM
Hello again!
d

Dusty Shapiro

09/28/2022, 4:16 PM
If I log that pod that errored, it simply says
Copy code
exec /usr/local/bin/dagster: exec format error
but pushing new image up with added dependency as well
Describing that pod only shows a successful pull of that image from our ECR but next step shows a back-off
d

daniel

09/28/2022, 4:19 PM
Hmm was this image possibly built on an M1 mac?
d

Dusty Shapiro

09/28/2022, 4:19 PM
Yes yes it was
d

daniel

09/28/2022, 4:20 PM
Can you try rebuilding with
docker build --platform linux/amd64
.
This sounds like a gross gotcha with different build archs
d

Dusty Shapiro

09/28/2022, 4:21 PM
Sure thing
Bad news: Not sure which of the two fixes above worked Good news: It works Thanks !
Next Docker build maybe I’ll remove the platform flag to see
d

daniel

09/28/2022, 4:43 PM
It probably would have failed in a new and exciting way without the dagster-k8s
but would have gotten farther
d

Dusty Shapiro

09/28/2022, 4:47 PM
Thanks again!
condagster 1
2 Views