Is it possible (and a good idea) to create a new class, called DynamicPartitionsDefinition, where I implement

get_partitions()

to get the list of partitions at runtime? In my use case, all my tables have a “dataset_id” column, that identify the source dataset used to generate the row. I want to have a partition per dataset_id, but the list is not known beforehand, so that I cannot use

StaticPartitionsDefinition

. Some details: • there would be O(10,000) partitions (or more) • I would also implement a Sensor to run the adequate partition job when needed • the list of partitions (ie the list of datasets) would be obtained at runtime by an API call to an external service (https://huggingface.co/api/datasets)

Hi Sylvain. Unfortunately Dagit doesn't support dynamic partitions at the moment, because calling out to user code to fetch the list of partition keys can be very slow and costly. We are considering supporting a similar runtime asset partitions feature, which adds a partition to an asset from within job runs rather than at definition time. Let us know if you have thoughts on this feature compared to dynamic partitions definitions! Not a great workaround, but for now you could define a function that makes the API call to fetch the list of partitions, then define a static partitions definition with the partitions list. Every time you reload your workspace, your static partitions definition would be updated.

Yes, changing the static partitions list could be an option... Beside it, I'm also wondering how I could manage the assets that disappear (in our context, when a dataset is deleted). I tried, and missing partitions that previously existed seem to just be ignored

Yep, as long as any deleted partition key is no longer part of the partitions definition, Dagster will just ignore it and not display that partition's data in Dagit