The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

are there plans to introduce memory usage graph per run? that would help a ton in debugging/monitoring huge assets.

I am not sure there are any plans to this today, but you could implement a workaround by adding memory size (or nrows, or w/e makes sense) as output metadata, and then dagster will plot it for you in the asset page

<https://github.com/dagster-io/hooli-data-eng-pipelines/blob/6d12f0e39fc16779054347eb0b7a529289aa5bf2/hooli_data_eng/assets/forecasting/__init__.py#L53-L78|https://github.com/dagster-io/hooli-data-eng-pipelines/blob/6d12f0e39fc16779054347[…]b0b7a529289aa5bf2/hooli_data_eng/assets/forecasting/__init__.py>

here is one example (see `Output(..., metadata...)` in the return)

and then the resulting plot