I’d like to bounce a feedback after the Roadmap Announcement:
Related to one of the claims
“Dagster has a reputation for being extremely powerful. However, the learning curve is still too steep.
One reason for this is that Dagster is actually three things rolled into one: it’s a scheduler, an operational asset catalog, and a data transformation tool.”
I think the Data/Asset catalog side of Dagster is lacking considerably in terms of its Usability. We are an organization that is heading towards Data maturity. Interactive orchestrator has brought lot of power with it and is becoming a GoTo tool for Developers and Business folks too, but the information derived from the “Data Catalog” side of Dagster is pretty crippled, either on purpose or as a side effect of Feature prioritizations.
If Dagster has to be a Data / Asset catalog, it needs a more mature UI for navigating and discovering assets and their relations and Data Lineage.
The static top down asset layout makes for a very unfriendly UX and a better ability to filter which asset lineage i want to see, is sorely missing with the current UI
Just as a Ref, DataHub Display of assets/Datasets and its expandable lineage.
https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:s3,longtail-c[…]llis&is_lineage_mode=true&schemaFilter=&start_time_millis
Small/Medium organizations simply cannot reach up to managing 2 overlapping systems in terms of pure man-power. So after running
Dagster Cluster, adding a data catalog like
DataHub just becomes an “overhead”
So keeping fingers crossed 🤞 for a better UI for asset catalog feature of Dagster