Hi everyone I have about 500 pipelines in a repository const

Question

Hi everyone I have about 500 pipelines in a repository constructed via yaml Every time I try to reload the repository it takes a large amount of time to reload Can anyone help me with this

owen · Answer

hi < Navneet Sajwan> it s unfortunately pretty buried in the documentation but it is possible to define your repository in a way such that not all jobs definitions are initialized on reload see options 2 3

Navneet Sajwan · Answer

Thanks for the response < owen> what is the beahaviour on reload in the 2 3 option

owen · Answer

hi < Navneet Sajwan> I believe if you use a pattern like what s shown under lazy loading repository then the jobs will only be constructed when needed like when you re about to launch a run or if you go to the page of a specific job

Navneet Sajwan · Answer

Actually I tried it in my local I did not see any significant improvement in repository loading times

Navneet Sajwan · Answer

It somehow worked in my kubernetes deployment When we click on pipeline earlier it used to take about 5 minutes to go from starting to started Now it takes less than 10 seconds for the same It s great but I still wonder how does it function in the backend < daniel> < rex> can you please refer to any docs how it worksin dagster