Hi, is there a way to cache resources or repositories in dagster? I have a repository that is huge and i want to cache it while dagster is reloading the repository. Background: I have a bad network and other apps running on the node so it is really not up to me to speed up the node itself. Instead i am look for a software solution in dagster api to cache the repository loading. Also, sometimes the load is so long fails the dagster daemon process because sensors become unresponsive. To get around this, i increased the time limit on heatbeat time of daemon processes. Any thoughts?

Ive never use this but you can construct the repo as a dict for lazy loading of jobs: https://docs.dagster.io/_apidocs/repositories#repositories

Hi @Metin Senturk, we do something like that but it’s not recommended as it’s basically editing internal job-related functions to use caching in the way. It may break between Dagster versions (and it did - which is fine because no one should mess with these internal functions). In our case, we defined a cached (

functools.cache

) version of

GraphDefinition._get_config_schema

. We re-defined a custom

<http://GraphDefinition.to|GraphDefinition.to>_job

to use the cached function. I’d recommend using separate repositories on separate gRPC-serving containers to balance the load instead of caching (we do that too).