Hey, I have a weekly partitioned multi asset that generates assets based on new files in an S3 bucket. Some of the outputs are not mandatory (

is_required=False

), because the provider might not put down the corresponding file for that week, while the rest of the files is expected to be there for each partition. The optional assets will have gaps in their partition history which in turn seems to turn on the “Stale” flag on the asset (even if the latest partition is green) which will propagate to downstream assets, making a big branch of my assets permanently stale even though we consider them fresh practically. Is there a way to make these optional partitioned assets look fresh if the latest partition was found? Or is it the opposite of what people would expect from a partitioned asset?

I’m not sure if this is the recommended way to go, but you might wanna look into

DynamicPartitions

. I have a similar use case with an asset that needs to be historized but is only irregularly updated. I believe it’s currently undocumented, but I found the code (incl. code example) to be very straightforward: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partition.py#L538 Alternatively, you could just materialize an empty object when the partition is not present.

Alternatively, you could just materialize an empty object when the partition is not present.

That’s what I’m looking into right now, but it feels “dirty”. I also lose some information about the missing partitions. It’s useful to know if one is missing, just don’t want it to make my asset stale if I already have a more recent partition.

DynamicPartitions

looks interesting though. I saw a few discussions earlier around the concept but I didn’t know it’s already implemented to some extent. Thanks for bringing it up!

Agree with Vinnie that dynamic partitions are a great application for this use case. We do have unreleased documentation out for this feature that will make it out this week: https://docs.dagster.io/master/concepts/partitions-schedules-sensors/partitions#dynamically-partitioned-assets You can go ahead and begin using the feature as it is already implemented, though it is experimental at the moment 🌈

Thanks for confirming!

@Andras Somi - if / when you try this out, I'm curious to hear how it goes for you

@sandy i'll let you know. I had to move on from this issue with a less than ideal solution of making the assets non-partitioned, but I plan to get back to the DynamicPartition thing.

@sandy I adapted this same pattern for what’s essentially a yearly budget asset. I want to keep it historized but it didn’t make sense to put it on a strict schedule; depending on the time of month it might not be updated at all or multiple times during a single day. Works pretty much as expected. The only “con” is there could be days/weeks missing from the list of partitions, but that’s to be expected. I haven’t needed any more information than it’s provided me out of the box.

would it be accurate to say that each partition corresponds to a time window, but those time windows aren't regularly spaced out?

Yep!