@Scout What’s the highest number partition in a MultiPartitionDefinition that I can have in Dagster?

The documentation does not explicitly state a maximum number of partitions that can be defined using a

MultiPartitionsDefinition

in Dagster. However, there are practical considerations to keep in mind when defining a large number of partitions. From the discussions and issues in the Dagster community, it is known that having too many partitions can lead to performance issues. For instance, there was an issue reported where Dagster became unresponsive when an asset was configured with a very high number of partitions (~90k partitions). This was due to the way

run_request_for_partition

iterated over all the partitions, which became expensive with such a large number. In another discussion, a user inquired about a reasonable limit for the number of partitions in a multi-dimension asset, mentioning a general recommendation of 10k partitions max. However, this discussion was left unanswered, indicating that there may not be a well-defined limit. Given these considerations, while there is no hard limit documented, it is advisable to be cautious when defining a large number of partitions and to test the performance implications in your specific Dagster deployment. If you encounter performance issues, you may need to adjust your partitioning strategy accordingly.