19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi everyone,
I have about 500 pipelines in a repos...
# ask-communityn
Navneet Sajwan
05/30/2022, 7:58 PMHi everyone,
I have about 500 pipelines in a repository , constructed via yaml . Every time I try to reload the repository, it takes a large amount of time to reload. Can anyone help me with this?
o
owen
05/31/2022, 4:41 PMhi @Navneet Sajwan, it's unfortunately pretty buried in the documentation, but it is possible to define your repository in a way such that not all jobs/definitions are initialized on reload (see options 2/3): https://docs.dagster.io/_apidocs/repositories#dagster.repository.
n
Navneet Sajwan
06/10/2022, 3:58 AMThanks, for the response @owen, what is the beahaviour on reload in the 2/3 option?
o
owen
06/10/2022, 10:44 PMhi @Navneet Sajwan, I believe if you use a pattern like what's shown under "lazy-loading repository", then the jobs will only be constructed when needed (like when you're about to launch a run or if you go to the page of a specific job)
n
Navneet Sajwan
06/14/2022, 4:39 AMActually, I tried it in my local, I did not see any significant improvement in repository loading times.
Navneet Sajwan
06/20/2022, 11:28 AMIt somehow worked in my kubernetes deployment. When we click on pipeline, earlier it used to take about 5 minutes to go from starting to started. Now, it takes less than 10 seconds for the same. It's great, but I still wonder how does it function in the backend. @daniel @rex can you please refer to any docs how it worksin dagster?
2 Views