Ivan Tsarev

05/04/2023, 2:08 PM
Hi folks! I wonder what is the best practices around using Dagster with dependency management tools like poetry. Our single-Definition project becomes a bit cluttery with several dozens of jobs and manual dependency management and I really want to reorganise into more pleasant way. So my initial idea is to split jobs into several definitions like "extractions", "business alerts", "export" etc and some "common" module with stuff like shared resources and utilities. But I have little understanding on what should be poetry projects to manage - every Definition-level folder to be able to split dependencies? And also how to properly make Cloud Dagster pick up dependencies from toml files?
:dagster-bot-responded-by-community: 1
:rainbow-daggy: 1


05/04/2023, 2:14 PM
You can check this github discussion: The pattern @Stephen Bailey showed in his post is essentially the same I’m following, and in my case poetry manages each code location/`Definitions` in its separate environment. It’s what’s felt the most natural, but of course you can always adapt it to what makes more sense for you.

Ivan Tsarev

05/04/2023, 2:37 PM
Thank you for link! I see one issue right now for implementing similar patter as Stephen proposes, because I'm using
approach and cannot create multiple "groups" of jobs inside one Locations. But still I found example repo within this discussion which also looks incredibly helpful.
By the way, is there any way to visually (at least) group jobs inside job list in dagit UI?
Or I probably should make them Assets beforehand, right?


05/04/2023, 4:33 PM
i guess it depends on the reason behind dividing jobs/assets into groups. if you are mostly looking for organizational purpose, i think asset group should be sufficient (best practice guide). but if you have to split because of dependency separation, then yeah you do need multiple code locations
in terms of multiple groups inside one code location, we currently don’t have the grouping capability for jobs, but we do support it for assets: UI looks like:

Ivan Tsarev

05/04/2023, 6:48 PM
Yep, it is 100% organisational for now. So I really should think about why I ended up with whole system organized with graph-based job instead of assets and what my actual reasoning behind it we we started to build our pipelines.