One thing I’ve struggled with, and have had some coworkers struggle with when trying to onboard them to dagster is all the ways to express config.
• Resource Config ( EG. when we define the configuration of a resource def)
• Config of resources (AKA ‘required resources’)
• Op config
• Run config (which seems to be pipeline config, something thats not really exposed much in 1.xx)
• Job Config
• input/output config
• config via code vs. via yaml
• config mappings
• sensor configs
• more, I may have missed some.
• Things that can be configured with @configured, .configured vs. others that cant (even if it doesn’t always make sense intuitively).
This isn’t to say this is all bad, obviously being able to configure all these different things in different ways gives a lot of power, but its also hard to know:
• With all these options, which should you use?
• How to differentiate them (EG: Many times I’ve gotten a a config type to resolve to None, when I thought I’d configured it.
Some sort of holistic documentation/guide on configuration may be a good idea, as well as some best practices. I’m still relatively new to the dagster project as a whole so I imagine this becomes second nature after using it for a bit longer. Just thought I’d point out a part of the learning curve that I and the team I work with have found too be a little steeper than expected.
Thanks for reading 🙂