Is it possible to define an asset that has different dependencies per partition that is dynamic?

Yes, it is possible to define an asset with dynamic partitions that have different dependencies per partition. The documentation describes the use of a

DynamicPartitionsDefinition

for cases where you don't know the set of partitions ahead of time when defining your assets. This allows for the addition of new partitions dynamically, such as when a new data file lands in a directory or when experimenting with a new set of hyperparameters. Furthermore, when a partitioned asset depends on another partitioned asset, each partition in the downstream asset depends on a partition or multiple partitions in the upstream asset. Dagster has default dependency rules that create the expected partition dependencies in most situations. However, users can override these by supplying a

PartitionMapping

to handle specific dependency scenarios, such as a rolling window situation where each daily partition of an asset depends on the prior three daily partitions of an upstream asset. For dynamic partitioning, you can define dependencies using sensors or other mechanisms to detect the presence of new partitions and trigger runs for those partitions. The

DynamicPartitionsDefinition

class accepts a

name

argument, representing the name of the partition set, and partition keys can be added and removed dynamically for a given partition set. In summary, Dagster provides the flexibility to define dynamic partitions with varying dependencies through the use of

DynamicPartitionsDefinition

and

PartitionMapping

.