Is it possible (and a good idea) to create a new c...
# ask-community
Is it possible (and a good idea) to create a new class, called DynamicPartitionsDefinition, where I implement
to get the list of partitions at runtime? In my use case, all my tables have a “dataset_id” column, that identify the source dataset used to generate the row. I want to have a partition per dataset_id, but the list is not known beforehand, so that I cannot use
. Some details: • there would be O(10,000) partitions (or more) • I would also implement a Sensor to run the adequate partition job when needed • the list of partitions (ie the list of datasets) would be obtained at runtime by an API call to an external service (
🤖 1
But it does not seem to be available for now blob sad I get
Copy code
dagster._core.errors.DagsterInvalidDefinitionError: Only static, time window, and multi-dimensional partitions are currently supported.
when instancing a DynamicPartitionsDefinition object
Hi Sylvain. Unfortunately Dagit doesn't support dynamic partitions at the moment, because calling out to user code to fetch the list of partition keys can be very slow and costly. We are considering supporting a similar runtime asset partitions feature, which adds a partition to an asset from within job runs rather than at definition time. Let us know if you have thoughts on this feature compared to dynamic partitions definitions! Not a great workaround, but for now you could define a function that makes the API call to fetch the list of partitions, then define a static partitions definition with the partitions list. Every time you reload your workspace, your static partitions definition would be updated.
Yes, changing the static partitions list could be an option... Beside it, I'm also wondering how I could manage the assets that disappear (in our context, when a dataset is deleted). I tried, and missing partitions that previously existed seem to just be ignored
Yep, as long as any deleted partition key is no longer part of the partitions definition, Dagster will just ignore it and not display that partition's data in Dagit
👍 1