Hi folks! I wonder what is the best practices around using Dagster with dependency management tools like poetry. Our single-Definition project becomes a bit cluttery with several dozens of jobs and manual dependency management and I really want to reorganise into more pleasant way. So my initial idea is to split jobs into several definitions like "extractions", "business alerts", "export" etc and some "common" module with stuff like shared resources and utilities. But I have little understanding on what should be poetry projects to manage - every Definition-level folder to be able to split dependencies? And also how to properly make Cloud Dagster pick up dependencies from toml files?

You can check this github discussion: https://github.com/dagster-io/dagster/discussions/8972 The pattern @Stephen Bailey showed in his post is essentially the same I’m following, and in my case poetry manages each code location/`Definitions` in its separate environment. It’s what’s felt the most natural, but of course you can always adapt it to what makes more sense for you.

Thank you for link! I see one issue right now for implementing similar patter as Stephen proposes, because I'm using

Definitions

approach and cannot create multiple "groups" of jobs inside one Locations. But still I found example repo https://github.com/yuhan/demo-my-dagster-project-with-multi-repos within this discussion which also looks incredibly helpful.

i guess it depends on the reason behind dividing jobs/assets into groups. if you are mostly looking for organizational purpose, i think asset group should be sufficient (best practice guide). but if you have to split because of dependency separation, then yeah you do need multiple code locations

Yep, it is 100% organisational for now. So I really should think about why I ended up with whole system organized with graph-based job instead of assets and what my actual reasoning behind it we we started to build our pipelines.