The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hi all, could someone have a look at this question: <https://softwareengineering.stackexchange.com/questions/444361/design-patterns-for-versioning-steps-across-data-workflow-pipelines>
It's not explicitly about dagster, but I'd love your opinions (and to be clear, I am using dagster)

In short, I'm finding it hard to keep a clean mental model of my dag over time. I get questions like:

1. Is the change in asset x due to the code change or the data change
2. Can you compare asset x as materialized last week to today
3. Can you compare assets ran with different configs
And it's making my mental model get very messy. Keen to get thoughts.

So... I have a dag, and it changes over time, and the assets change over time, and the inputs change, but sometimes they get run with different configs, and sometimes an asset as materialized in a previous run should be compared to the current run...

It gets messy

Wondering if you’ve seen this guide: <https://docs.dagster.io/guides/dagster/asset-versioning-and-caching#asset-versioning-and-caching>