Sylvain Lesage01/06/2023, 2:33 PM
to get the list of partitions at runtime? In my use case, all my tables have a “dataset_id” column, that identify the source dataset used to generate the row. I want to have a partition per dataset_id, but the list is not known beforehand, so that I cannot use
. Some details: • there would be O(10,000) partitions (or more) • I would also implement a Sensor to run the adequate partition job when needed • the list of partitions (ie the list of datasets) would be obtained at runtime by an API call to an external service (https://huggingface.co/api/datasets)
when instancing a DynamicPartitionsDefinition object
dagster._core.errors.DagsterInvalidDefinitionError: Only static, time window, and multi-dimensional partitions are currently supported.
claire01/06/2023, 6:39 PM
Sylvain Lesage01/06/2023, 6:49 PM
claire01/06/2023, 6:50 PM