Posted this in #dagster-cloud , hoping to get assistance. Thanks! https://dagster.slack.com/archives/C02LJ7G0LAZ/p1684371830418209

@Ashwin Kamath I have also been seeing this error with dynamic partitions lately. In my case though, there is nothing tricky about them - they are supposed to be the same dynamic partitions for both assets, and I still get this error on the downstream asset.

Thanks for confirming, the strange thing about this is that the stack trace seems to be using the default partition mapping class instead of my custom one. Unclear to me how this code is called in the first place as its all within the cloud agent

Hey Ashwin, would you mind sharing your asset defs and the partition mapping def? Does this error happen locally too or just on cloud?

Hi @claire, yes, it seems to be a bug of some sort (unless I am just missing something - entirely possible!). I can materialize the downstream assets one-by-one, but when I try to materialize a range of them I get an error about missing upstream assets. The partitions are being created by a schedule. I’m not sure how to send the partitions def without sending a lot of code…

Is this happening on cloud or locally?

Locally

My issue occurs on cloud and local, sure can try to create a simple repro

Yeah, you can override it by doing something like this:

@asset(
    partitions_def=partitions_def,
    ins={"asset1": AssetIn(partition_mapping=<Your partition mapping def here>},
)
def downstream_asset1(asset1):
    return 2

I have that set

Got it. Once you send me the repro I will take a look

This is the partition mapping, I will try to get you a repro in the next day thanks

Hey Ashwin. Appreciate the detailed repro here, I was able to repro this. Unfortunately writing your own custom partition mapping is unsupported at this time. Subsequently for asset backfills, we ignore partition mappings that aren't built-in as part of dagster, which is why your partition mapping is being ignored here. This kind of issue is a known pain point in our existing system--supporting custom partition mappings requires a function call over gRPC to fetch partition dependencies that would make things like auto-materialization potentially prohibitively slow, which is why we don't support this behavior. But it is annoying that users can't define their own partition mappings like in your case. I would recommend filing an issue for this in the meantime. We have a similar issue here: https://github.com/dagster-io/dagster/issues/13139 One possible alternative would be to implement your own custom subclass of the

TimeWindowPartitionMapping

TimeWindowPartitionsDefinition

, and use that as your upstream asset partitions def instead of a

DynamicPartitionsDefinition

.

Thanks for the response, that is unfortunate. I am in need of a "Business Days Partition" and Dynamic seemed to be the easiest approach. If I were to implement a custom subclass of TimeWindowPartitionMapping, would that get recognized during asset backfills? Seems to me it may run into the same issue, or am I not understanding things correctly?

Hey Ashwin, yeah, a custom subclass of TimeWindowPartitionMapping should not be recognized during the asset backfills. I had a typo in my message--instead of subclassing

TimeWindowPartitionMapping

, i meant

TimeWindowPartitionsDefinition

. I don't have personal experience doing something like this, but one possible implementation strategy would have an "exclude_days" arg in the constructor.

@claire - Hmm following up here, either way works for me.

exclude_days

or dynamic partition mapping. I do need something reasonably quickly so will try to dig deeper into the dagster internals