One major deterrent for myself from using dagster is the lack of quality resources to get up and running past the cereal DAGs. I've been able to pick up airflow much faster not only because of the community, but because there are ample amounts of udemy courses, youtube videos and medium articles detailing most aspects of the tool. For dagster, I've found very little in this realm. While it seems to do a lot of amazing things, I can't seem to overcome the hurdle of learning the tool. I can't imagine others aren't in the same boat as myself

This is really important feedback. I too found Dagster very difficult to grok and while I'm comfortable with the vast majority of it, there are parts I haven't touched and don't know if I want to, like assets, simply because it is so much to learn, and it's specific to Dagster. One specific frustration I had was that there were several times where I spun my wheels trying to figure something out, like good ways to organize repos or using docker-compose for local development, only to later find code examples somewhere in GitHub. These gave me some guidance, but I also wouldn't have sought them out or known how to find them on my own. Point being, There's considerable documentation and code examples, which is great, but the scope of the product is so large that it's really hard to grok it all / onboard in a coherent way. I have felt like I'm using 100% of my engineering brain capacity for the last month of getting our deployment out

I think a lot of people (myself included) want to love this product

I compare it to dbt, where the mental load builds over time — first you learn to build models (the language). Then you learn to do dev/prod and resource targeting. Then you learn to deploy it. I could be misremembering the experience, but i felt like it was much more gradual. With Dagster I feel like I have to think about everything right up front, because I'm spinning up daemons and dagit and wrapping Python code in Dagster decorators. I'm not sure what the answer is, but it's a challenge for sure

I think maybe it is just a really steep learning curve. I agree that there is a lot that is specific to Dagster. For example, the concept of an op & graph. I'm not entirely sure how they differ or what their clear definitions are

While I’m working on applied research area that hard engineering tasks are not required, my colleague at the data engineering position says that dagster have to have

example repositories for major use-cases

based on minimum viable product repository which is introduced in #dagster-showcase . I do understand that dagster’s version is 0.x.y and it is not easy to fix example shocases for every use cases because major version changes make these examples useless… However, it might be still possible to build example repositories for

important use cases

that dagster’s concept is distinguished among other similar tools.

Hey @Atsushi Saito @Stephen Bailey and @Alec Ryan thanks for giving this direct and essential feedback. We strive to have a “progressive disclosure of complexity” in our product and documentation but it’s clear from your commentary that we have work to do.

No problem, Nick. After working with dagster for a little over a week now, I feel like I am starting to figure it out. I will say that there are many times where I feel the documentation is lacking, especially with SDAs. As a beginner, I think a more robust tutorial section (accompanied with videos perhaps) would be incredibly beneficial.

SDA documentation more immature relative to the rest of the system. Very useful context to know that that is what you are looking at.

Yeah, I would say documentation of all of the "productionizing" materials is surprisingly good. Lots of thorough answers to specific questions. The challenge I have is having to look at the documentation so much 🙂 But i think its a situation of "simplicity on the far side of complexity", such that, now that the deployment is set up, my data scientists / analysts should be able to focus on just coding in a way they wouldn't be able to otherwise