The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Not sure if here is the right place to ask: We deploy OSS dagster on kubernetes, after upgraded to 1.1.6 (from 1.0.17) we see the `Definitions reloaded!` keep poping up every ~20 seconds. Rolling back to 1.1.5 resolve the issue, any idea why?

<@U039VKAHCAY> and I are thinking this might have been fixed by <@UM49TQ8EB>

does it require all code servers update to 1.1.6 libs as well?

I tried rolling dagit between 1.1.5 and 1.1.6 and keep seeing this happening in the later version though (yesterday)

Both the bug and the fix were introduced in `1.1.6` so any of the behavior seeing now might be new.  I fundamentally changed the query pattern to compare timestamps of the loaded code locations.  By any chance are you running multiple dagit instances?

If so, it could be that the repeated queries are hitting different dagit instances and subsequent queries appear to be getting newer timestamps (randomly)

we only have one but with 2 replica for both rw and readonly instances.

is there any log i can get and send to you for debugging?

Sorry, I’m not sure what the replicas are… is this for your dagster storage (e.g. database replicas)?    Trying to figure out if you have multiple dagit webservers running, that might be holding different timestamps

yes, there are two pods running for dagit

so the pods in kubernetes looks like this:
```dagster-dagit-67d7dfb8c7-27sjd                                    1/1     Running     0          14h
dagster-dagit-67d7dfb8c7-g7qg2                                    1/1     Running     0          14h```

so the helm values for dagit looks like this:
```dagit:
  replicaCount: 2
  image:
    # When a tag is not supplied for a Dagster provided image,
    # it will default as the Helm chart version.
    repository: "<http://docker.io/dagster/dagster-celery-k8s|docker.io/dagster/dagster-celery-k8s>"
    tag: "1.1.6"
    pullPolicy: Always```

`replicaCount: 2` will results in two dagit webserver instances

I think I have a solution for this: <https://github.com/dagster-io/dagster/pull/11100>

Can probably get this out for `1.1.7`, slated to go out on Thursday

<@UM49TQ8EB> I saw this get reverted, in this case shall we update to 1.1.7?

Hi Pang… yeah, that PR had some issues that we identified during our release process so we had to revert it.  I put up <https://github.com/dagster-io/dagster/pull/11173>, which is my second take at it.  I think it will go out in the next release `1.1.8`, which probably won’t happen for another couple weeks.