https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
a

Alec Ryan

04/14/2022, 9:11 PM
One major deterrent for myself from using dagster is the lack of quality resources to get up and running past the cereal DAGs. I've been able to pick up airflow much faster not only because of the community, but because there are ample amounts of udemy courses, youtube videos and medium articles detailing most aspects of the tool. For dagster, I've found very little in this realm. While it seems to do a lot of amazing things, I can't seem to overcome the hurdle of learning the tool. I can't imagine others aren't in the same boat as myself
s

Stephen Bailey

04/15/2022, 3:55 PM
This is really important feedback. I too found Dagster very difficult to grok and while I'm comfortable with the vast majority of it, there are parts I haven't touched and don't know if I want to, like assets, simply because it is so much to learn, and it's specific to Dagster. One specific frustration I had was that there were several times where I spun my wheels trying to figure something out, like good ways to organize repos or using docker-compose for local development, only to later find code examples somewhere in GitHub. These gave me some guidance, but I also wouldn't have sought them out or known how to find them on my own. Point being, There's considerable documentation and code examples, which is great, but the scope of the product is so large that it's really hard to grok it all / onboard in a coherent way. I have felt like I'm using 100% of my engineering brain capacity for the last month of getting our deployment out
plus1 4
a

Alec Ryan

04/15/2022, 3:56 PM
I think a lot of people (myself included) want to love this product
🙌 1
s

Stephen Bailey

04/15/2022, 4:00 PM
I compare it to dbt, where the mental load builds over time — first you learn to build models (the language). Then you learn to do dev/prod and resource targeting. Then you learn to deploy it. I could be misremembering the experience, but i felt like it was much more gradual. With Dagster I feel like I have to think about everything right up front, because I'm spinning up daemons and dagit and wrapping Python code in Dagster decorators. I'm not sure what the answer is, but it's a challenge for sure
💯 1
Yes! That was my exact feeling when I first tried it out - I want to love this but it feels too hard
When I compared with Airflow and Prefect, though, I decided it was worth the effort and have not been disappointed
a

Alec Ryan

04/15/2022, 4:06 PM
I think maybe it is just a really steep learning curve. I agree that there is a lot that is specific to Dagster. For example, the concept of an op & graph. I'm not entirely sure how they differ or what their clear definitions are
I feel like I get it more and more every day, so maybe that is just it. It takes time to learn.
a

Atsushi Saito

04/20/2022, 3:46 PM
While I’m working on applied research area that hard engineering tasks are not required, my colleague at the data engineering position says that dagster have to have
example repositories for major use-cases
based on minimum viable product repository which is introduced in #dagster-showcase . I do understand that dagster’s version is 0.x.y and it is not easy to fix example shocases for every use cases because major version changes make these examples useless… However, it might be still possible to build example repositories for
important use cases
that dagster’s concept is distinguished among other similar tools.
I also use
kedro
which is popular among data scientists and researchers, and
Prefect
provides the way how to convert
kedro
pipeline source codes into Prefect source code. This strategy of Prefect is effective and several corporations started to use Prefect. https://kedro.readthedocs.io/en/stable/deployment/prefect.html
Because dagster has AirFlow integrations, enhancing the integrations could increase the number of users which is currently using AirFlow. Then, YouTube tutorial video and medium blog posts would be produced than the current status.
s

schrockn

04/22/2022, 5:48 PM
Hey @Atsushi Saito @Stephen Bailey and @Alec Ryan thanks for giving this direct and essential feedback. We strive to have a “progressive disclosure of complexity” in our product and documentation but it’s clear from your commentary that we have work to do.
🎉 1
a

Alec Ryan

04/24/2022, 1:18 PM
No problem, Nick. After working with dagster for a little over a week now, I feel like I am starting to figure it out. I will say that there are many times where I feel the documentation is lacking, especially with SDAs. As a beginner, I think a more robust tutorial section (accompanied with videos perhaps) would be incredibly beneficial.
s

schrockn

04/24/2022, 1:35 PM
SDA documentation more immature relative to the rest of the system. Very useful context to know that that is what you are looking at.
s

Stephen Bailey

04/24/2022, 6:24 PM
Yeah, I would say documentation of all of the "productionizing" materials is surprisingly good. Lots of thorough answers to specific questions. The challenge I have is having to look at the documentation so much 🙂 But i think its a situation of "simplicity on the far side of complexity", such that, now that the deployment is set up, my data scientists / analysts should be able to focus on just coding in a way they wouldn't be able to otherwise