https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alex Kan

02/15/2023, 8:43 PM
Is there a way to mark partitions as Skipped or Succeeded? • The use case here is that I want to re-partition an asset, but don’t want to have to backfill. I also don’t want partitions that previously ran to be considered “missing” • ex. Imagine I have partitions defining ranges 0-99, 100-199 …, but want to change the partitions to represent 0-9, 10-19 … Related, is there a max # of partitions that an asset could have (hopefully without causing perf regressions in Dagit)
o

owen

02/15/2023, 10:11 PM
hi @Alex Kan! There's currently no direct way to mark partitions with a given status, but the general recommendation here is just to write a dummy op that yields a bunch of AssetMaterialization events with the partition keys you want to show as "filled in" (i.e.
yield AssetMaterialization(asset_key=AssetKey("my_key"), partition="0-9")
)
👌 1
re: partitions perf I think you should be fine until around 20k or so as far as I'm aware (cc @claire)
c

claire

02/15/2023, 10:16 PM
Yep, I think you should be okay with 20K partitions
a

Alex Kan

02/15/2023, 10:27 PM
Ah that’s smart! Could easily write a utility script to yield those events. Was playing around earlier and bricked Dagit with 70k partitions and should def be able to get that down. Thanks all!
blob salute 1
o

owen

02/15/2023, 10:28 PM
(also nice to meet you not outside murraccis 😄)
a

Alex Kan

02/15/2023, 10:28 PM
Was that the first Dagster celebrity sighting? 😛
blob joy 1
👌 1
y

Yeachan Park

04/17/2023, 11:26 AM
@owen I assume this works only for the asset partition view. Is there a way we can mark the corresponding job partitions (for the assets) as succeeded as well?
5 Views