Joram Sbrisny
02/21/2024, 8:37 PMScout
02/21/2024, 8:37 PMDynamicPartitionsDefinition
. This allows you to define a partition set where the partitions are not known ahead of time and can be added or removed dynamically. Here's an example from the documentation: python from dagster import DynamicPartitionsDefinition, asset images_partitions_def = DynamicPartitionsDefinition(name="images") @asset(partitions_def=images_partitions_def) def images(context): ...
In this case, you can use a sensor to detect the presence of new data and trigger a run for the new partition. 2. Dynamic Output: If you have a scenario where an asset computation might result in a dynamic number of outputs, you can use DynamicOutput
within your ops. However, this is more relevant to the execution of ops within a job and not directly related to the creation of assets. It's important to note that while Dagster supports dynamic partitioning, the creation of entirely new assets dynamically (i.e., defining new asset functions or classes at runtime) is not a pattern that is explicitly supported according to the documentation. Instead, Dagster encourages the use of partitions to handle dynamic datasets within a single asset definition. If you need further guidance on how to implement dynamic assets for your specific use case, please provide more details, and I can help you navigate the documentation to find the most suitable solution.