https://dagster.io/ logo
Join Slack
Powered by
hi im getting an "Unexpected Error" in dagit, depl...
# deployment-kubernetes
b
hi im getting an "Unexpected Error" in dagit, deployed on k8s when trying to refresh my repository.:
Copy code
Operation name: ReloadRepositoryLocationMutation

Message: 'JobType'

Path: ["reloadRepositoryLocation"]

Locations: [{"line":2,"column":3}]

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/executor.py", line 452, in resolve_or_error
    return executor.execute(resolve_fn, source, info, **args)
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/executors/sync.py", line 16, in execute
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/schema/roots.py", line 504, in mutate
    graphene_info.context.reload_repository_location(location_name)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/context.py", line 63, in reload_repository_location
    new_location = RepositoryLocation.from_handle(new_handle)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 148, in from_handle
    return GrpcServerRepositoryLocation(repository_location_handle)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 330, in __init__
    self._handle.client, self._handle,
  File "/usr/local/lib/python3.7/site-packages/dagster/api/snapshot_repository.py", line 58, in sync_get_streaming_external_repositories_grpc
    for chunk in external_repository_chunks
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 280, in deserialize_json_to_dagster_namedtuple
    check.str_param(json_str, "json_str"), whitelist_map=_WHITELIST_MAP
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 292, in _deserialize_json_to_dagster_namedtuple
    return _unpack_value(seven.json.loads(json_str), whitelist_map=whitelist_map)
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 252, in _unpack_value
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 252, in <dictcomp>
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 238, in _unpack_value
    return [_unpack_value(i, whitelist_map) for i in val]
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 238, in <listcomp>
    return [_unpack_value(i, whitelist_map) for i in val]
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 252, in _unpack_value
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 252, in <dictcomp>
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 267, in _unpack_value
    return getattr(whitelist_map["types"]["enum"][name], member)
a
is dagit running at the same version as your user code deployments?
b
hmm i can check that but it should be. was there a recent release?
s
yup we had a release last thursday
you may be using the 
:latest
tag somewhere which is why your gRPC server is running on 0.10.0
b
okay, but im just using this for the 
dagit
config
Copy code
image:
    repository: "dagster/k8s-dagit-example"
    tag: "latest"
    pullPolicy: Always
so im guessing thats not at 0.10.0 ?
yeah, it seems like all my jobs stopped running last thursday because they couldn't connect to the gRPC server
s
the 
latest
version of 
k8s-dagit-example
is 0.10.0, when was the last time your dagit pod restarted? if you try killing it, it’ll pick up the latest version on restart
going forward, it might be a good idea to pin your images to a specific tag (like 
0.10.0
)
There is a bit of a migration process for 
0.10.0
however, so if you don’t want to go through it now you can also set both your gRPC server and dagit images to a previous version https://github.com/dagster-io/dagster/blob/master/MIGRATION.md
b
okay, i think i've gotten through most of these steps but im stuck on a helm error
Copy code
Error: Deployment.apps "dagster-dagit" is invalid: [spec.template.spec.containers[0].livenessProbe.tcpSocket: Forbidden: may not specify more than 1 handler type, spec.template.spec.containers[0].startupProbe.tcpSocket: Forbidden: may not specify more than 1 handler type]
a
@rex
b
here are my full helm values if that's helpful
dagster-values.yaml
👍 1
r
cc @bob looks like a liveness probe issue
👀 1
b
thank you for all your help! i hope this is useful in someway.
a
for sure, thanks for the report and follow-up. Hopefully we can get this fixed and in a release later this week.
b
we recently changed the default probe handler from 
tcpSocket
to 
httpGet
on “/dagit_info”. i think the issue is that your setting 
dagit.livenessProbe.tcpSocket
is conflicting with the 
httpGet
probe handler one fix would be to remove your 
tcpSocket
object from 
dagit.livenessProbe
and 
dagit.startupProbe
alternatively, if you want to keep using the 
tcpSocket
probe handler, then i think you can set 
dagit.livenessProbe.httpGet: null
(and `dagit.startupProbe.httpGet: null`) to get rid of the error
r
just as a sanity check, did you update your helm repo index? Running 
helm search repo dagster --versions
should show the latest version (0.10.0)
Copy code
~/elementl/dagster master
dagster-3.7.4 ❯ helm search repo dagster --versions
NAME           	CHART VERSION	APP VERSION	DESCRIPTION
dagster/dagster	0.10.0       	0.10.0     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.22       	0.9.22     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.21       	0.9.21     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.20       	0.9.20     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.19       	0.9.19     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.18       	0.9.18     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.17       	0.9.17     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.16       	0.9.16     	Dagster is a system for building modern data ap...
7 Views
Open in Slack
PreviousNext
Powered by