I did a POC with Dagster 3 years ago but unfortunately we didn’t have the resources at the time to migrate our existing pipelines. I’m now working on a greenfield data pipeline architecture and I really want dagster to work for us! Looks like a lot has changed (and I’ve forgotten a lot), so I’m basically starting from scratch. I’m working through the tutorial from the start and finding it to be a bit awkward. I thought I’d record my impressions as a new user with fresh eyes in hopes that it could be improved. I’m also seeing this from the perspective of a future Dagster cloud user who would mostly be executing dags through dagit rather than from the command line. Here are some of my initial observations: • The first tutorial task of defining an asset makes sense. However, when it moves on to defining a more complex asset, it’s not clear that we’re supposed to close the dagit session from the first page. When we start up the new dagit session for

complex_asset_graph.py

, I can no longer see the original

serial_asset_graph.py

. I’ve now finished all of the sections under the “Tutorials” section, and it’s still not clear to me how I would see and manage both of these jobs simultaneously within dagit. Looks like it might be explained in the “Concepts” section, but that’s a lot more to go through. • The “Testing Assets” section instructs us to execute

pytest test_complex_asset_graph.py

, but it never explicitly told us what code to put in that file. If we assume it’s the code right above that command, then that code needs to include an

import

of all of the assets referenced in the test file, otherwise it raises errors. Furthermore, it never explicitly says you have to install

pytest

. • After the “Assets” section, it goes on the “Ops and Jobs” section, which feels mostly like an exact repeat of the “Assets” section, just with different decorators. I don’t know why I would use ops and jobs over assets or vice-versa, and I’m more confused now about what purpose of assets were in the first place. At the end of the main “Tutorial” section, I don’t really have a good big picture understanding of dagster and how all of the pieces fit together. I’m sure it will become more clear as dig into the Concepts section, but that has a ton more content and just browsing it appears to be more of a reference doc than a tutorial or walkthrough of the basics.

Hi @Sterling Paramore really appreciate this thorough feedback! I think your takes are very on point and it is definitely something we've been wanting to improve. cc @erin who's actively working on the docs improvements right now.

This is really excellent feedback - thanks for taking the time to share! I've been looking at ways to improve these tutorials and this will be really helpful for that effort.

I think what would really help me is if there was a tutorial that described developing the hacker news example (or maybe even something more simplified, while still being somewhat realistic). I’d like to see the steps involved in building it, rather than having it already built and having to decompose it all myself.