https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
b

Brian Abelson

01/20/2021, 8:44 PM
hi im getting an "Unexpected Error" in dagit, deployed on k8s when trying to refresh my repository.:
Copy code
Operation name: ReloadRepositoryLocationMutation

Message: 'JobType'

Path: ["reloadRepositoryLocation"]

Locations: [{"line":2,"column":3}]

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/executor.py", line 452, in resolve_or_error
    return executor.execute(resolve_fn, source, info, **args)
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/executors/sync.py", line 16, in execute
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/schema/roots.py", line 504, in mutate
    graphene_info.context.reload_repository_location(location_name)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/context.py", line 63, in reload_repository_location
    new_location = RepositoryLocation.from_handle(new_handle)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 148, in from_handle
    return GrpcServerRepositoryLocation(repository_location_handle)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 330, in __init__
    self._handle.client, self._handle,
  File "/usr/local/lib/python3.7/site-packages/dagster/api/snapshot_repository.py", line 58, in sync_get_streaming_external_repositories_grpc
    for chunk in external_repository_chunks
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 280, in deserialize_json_to_dagster_namedtuple
    check.str_param(json_str, "json_str"), whitelist_map=_WHITELIST_MAP
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 292, in _deserialize_json_to_dagster_namedtuple
    return _unpack_value(seven.json.loads(json_str), whitelist_map=whitelist_map)
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 252, in _unpack_value
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 252, in <dictcomp>
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 238, in _unpack_value
    return [_unpack_value(i, whitelist_map) for i in val]
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 238, in <listcomp>
    return [_unpack_value(i, whitelist_map) for i in val]
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 252, in _unpack_value
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 252, in <dictcomp>
    unpacked_val = {key: _unpack_value(value, whitelist_map) for key, value in val.items()}
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 267, in _unpack_value
    return getattr(whitelist_map["types"]["enum"][name], member)
a

alex

01/20/2021, 8:47 PM
is dagit running at the same version as your user code deployments?
b

Brian Abelson

01/20/2021, 8:52 PM
hmm i can check that but it should be. was there a recent release?
s

sashank

01/20/2021, 8:54 PM
yup we had a release last thursday
you may be using the
:latest
tag somewhere which is why your gRPC server is running on 0.10.0
b

Brian Abelson

01/20/2021, 8:58 PM
okay, but im just using this for the
dagit
config
Copy code
image:
    repository: "dagster/k8s-dagit-example"
    tag: "latest"
    pullPolicy: Always
so im guessing thats not at 0.10.0 ?
yeah, it seems like all my jobs stopped running last thursday because they couldn't connect to the gRPC server
s

sashank

01/20/2021, 9:01 PM
the
latest
version of
k8s-dagit-example
is 0.10.0, when was the last time your dagit pod restarted? if you try killing it, it’ll pick up the latest version on restart
going forward, it might be a good idea to pin your images to a specific tag (like
0.10.0
)
There is a bit of a migration process for
0.10.0
however, so if you don’t want to go through it now you can also set both your gRPC server and dagit images to a previous version https://github.com/dagster-io/dagster/blob/master/MIGRATION.md
b

Brian Abelson

01/20/2021, 10:51 PM
okay, i think i've gotten through most of these steps but im stuck on a helm error
Copy code
Error: Deployment.apps "dagster-dagit" is invalid: [spec.template.spec.containers[0].livenessProbe.tcpSocket: Forbidden: may not specify more than 1 handler type, spec.template.spec.containers[0].startupProbe.tcpSocket: Forbidden: may not specify more than 1 handler type]
a

alex

01/20/2021, 10:53 PM
@rex
b

Brian Abelson

01/20/2021, 10:54 PM
here are my full helm values if that's helpful
👍 1
r

rex

01/20/2021, 10:54 PM
cc @bob looks like a liveness probe issue
👀 1
b

Brian Abelson

01/20/2021, 10:55 PM
thank you for all your help! i hope this is useful in someway.
a

alex

01/20/2021, 10:56 PM
for sure, thanks for the report and follow-up. Hopefully we can get this fixed and in a release later this week.
b

bob

01/20/2021, 11:07 PM
we recently changed the default probe handler from
tcpSocket
to
httpGet
on “/dagit_info”. i think the issue is that your setting
dagit.livenessProbe.tcpSocket
is conflicting with the
httpGet
probe handler one fix would be to remove your
tcpSocket
object from
dagit.livenessProbe
and
dagit.startupProbe
alternatively, if you want to keep using the
tcpSocket
probe handler, then i think you can set
dagit.livenessProbe.httpGet: null
(and `dagit.startupProbe.httpGet: null`) to get rid of the error
r

rex

01/20/2021, 11:09 PM
just as a sanity check, did you update your helm repo index? Running
helm search repo dagster --versions
should show the latest version (0.10.0)
Copy code
~/elementl/dagster master
dagster-3.7.4 ❯ helm search repo dagster --versions
NAME           	CHART VERSION	APP VERSION	DESCRIPTION
dagster/dagster	0.10.0       	0.10.0     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.22       	0.9.22     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.21       	0.9.21     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.20       	0.9.20     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.19       	0.9.19     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.18       	0.9.18     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.17       	0.9.17     	Dagster is a system for building modern data ap...
dagster/dagster	0.9.16       	0.9.16     	Dagster is a system for building modern data ap...
3 Views