The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hello. If I have an asset which depends on a partitioned asset, how can I make declare that the downstream asset depends on all of the upstream assets. Basically my downstream asset is aggregating all of the partitions back into one dataframe. But it always ends up stale after runs because it gets materialised when only some of the upstream partitions have finished. How do I make Dagster wait to materialise the downstream asset until all of the upstream asset's partitions are fresh?

hi <@U05749BHDJ5>! by default, dagster should assume that an unpartitioned downstream asset depends on all partitions of the upstream asset. how are you executing these assets?

yeah, strange, i may have spoken too soon, let me play with it for a few more mins :S

Okay, yes I was confused because when you click on the backfill and it shows you the queued runs, they weren't all there. But downstream runs get added later as the backfill progresses.

The run status still shows missing partitions though even though the backfill completed successfully though, which is a bit odd.