https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
p

Pang Wu

12/13/2022, 5:44 PM
Not sure if here is the right place to ask: We deploy OSS dagster on kubernetes, after upgraded to 1.1.6 (from 1.0.17) we see the
Definitions reloaded!
keep poping up every ~20 seconds. Rolling back to 1.1.5 resolve the issue, any idea why?
r

rex

12/13/2022, 5:51 PM
@dish
d

dish

12/13/2022, 5:51 PM
@Marco Salazar and I are thinking this might have been fixed by @prha
(Though I thought the fix was in 1.1.6)
p

Pang Wu

12/13/2022, 5:52 PM
does it require all code servers update to 1.1.6 libs as well?
I tried rolling dagit between 1.1.5 and 1.1.6 and keep seeing this happening in the later version though (yesterday)
p

prha

12/13/2022, 5:56 PM
Both the bug and the fix were introduced in
1.1.6
so any of the behavior seeing now might be new. I fundamentally changed the query pattern to compare timestamps of the loaded code locations. By any chance are you running multiple dagit instances?
If so, it could be that the repeated queries are hitting different dagit instances and subsequent queries appear to be getting newer timestamps (randomly)
p

Pang Wu

12/13/2022, 6:50 PM
we only have one but with 2 replica for both rw and readonly instances.
is there any log i can get and send to you for debugging?
p

prha

12/13/2022, 8:20 PM
Sorry, I’m not sure what the replicas are… is this for your dagster storage (e.g. database replicas)? Trying to figure out if you have multiple dagit webservers running, that might be holding different timestamps
p

Pang Wu

12/13/2022, 10:14 PM
yes, there are two pods running for dagit
so the pods in kubernetes looks like this:
Copy code
dagster-dagit-67d7dfb8c7-27sjd                                    1/1     Running     0          14h
dagster-dagit-67d7dfb8c7-g7qg2                                    1/1     Running     0          14h
so the helm values for dagit looks like this:
Copy code
dagit:
  replicaCount: 2
  image:
    # When a tag is not supplied for a Dagster provided image,
    # it will default as the Helm chart version.
    repository: "<http://docker.io/dagster/dagster-celery-k8s|docker.io/dagster/dagster-celery-k8s>"
    tag: "1.1.6"
    pullPolicy: Always
replicaCount: 2
will results in two dagit webserver instances
p

prha

12/13/2022, 10:45 PM
I think I have a solution for this: https://github.com/dagster-io/dagster/pull/11100
Can probably get this out for
1.1.7
, slated to go out on Thursday
p

Pang Wu

12/13/2022, 10:47 PM
Awesome! Thank you Phil!
cc @Saurabh Tripathi
@prha I saw this get reverted, in this case shall we update to 1.1.7?
p

prha

12/16/2022, 5:24 PM
Hi Pang… yeah, that PR had some issues that we identified during our release process so we had to revert it. I put up https://github.com/dagster-io/dagster/pull/11173, which is my second take at it. I think it will go out in the next release
1.1.8
, which probably won’t happen for another couple weeks.
p

Pang Wu

12/16/2022, 5:24 PM
gotcha, thanks for the update!