I’d like to bounce a feedback after the Roadmap An...
# dagster-feedback
I’d like to bounce a feedback after the Roadmap Announcement: Related to one of the claims “Dagster has a reputation for being extremely powerful. However, the learning curve is still too steep. One reason for this is that Dagster is actually three things rolled into one: it’s a scheduler, an operational asset catalog, and a data transformation tool.” I think the Data/Asset catalog side of Dagster is lacking considerably in terms of its Usability. We are an organization that is heading towards Data maturity. Interactive orchestrator has brought lot of power with it and is becoming a GoTo tool for Developers and Business folks too, but the information derived from the “Data Catalog” side of Dagster is pretty crippled, either on purpose or as a side effect of Feature prioritizations. If Dagster has to be a Data / Asset catalog, it needs a more mature UI for navigating and discovering assets and their relations and Data Lineage. The static top down asset layout makes for a very unfriendly UX and a better ability to filter which asset lineage i want to see, is sorely missing with the current UI Just as a Ref, DataHub Display of assets/Datasets and its expandable lineage. https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:s3,longtail-c[…]llis&is_lineage_mode=true&schemaFilter=&start_time_millis Small/Medium organizations simply cannot reach up to managing 2 overlapping systems in terms of pure man-power. So after running Dagster Cluster, adding a data catalog like DataHub just becomes an “overhead” So keeping fingers crossed 🤞 for a better UI for asset catalog feature of Dagster
thanks 1
👍 1
thanks for the feedback! we are actively exploring ideas to make this exact experience better. cc @josh for better asset organization, collapsible asset group in lineage view, etc.
Or https://www.meetup.com/de-DE/vienna-data-engineering-meetup/events/293990442/ OpenMetadatas sandbox as a 2nd inspiration for how a real data catalogs UI might look like