https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Maurício Barbosa

02/15/2024, 6:06 PM
Hello, I am consuming an external API to return a list of documents to process, and for each document that is returned, I am adding a new partition to a DynamicPartitionMapping in the following way: {config_id}_{doc_id}, where the config_id is an id that identifies the type of file and the doc_id is the file id. In this scenario, would it be ideal to use MultiPartitionsDefinition, or would it be better to continue using a dynamic partition where the partition uses a separator in its own name? My problem involves downloading each file, but I want to create different assets that process in specific configurations, for example: Asset A: has partitions {config_id}__{doc_id} and downloads each file_ _Asset B: has partitions X__{doc_id} and only processes the files downloaded by A that had config_id = X At the moment, I am creating a dynamic partition definition with the string {config_id}__{doc_id} for asset A, and also creating another definition for Asset B with X__{doc_id}, but I am having difficulties in how to define the partition_mapping.
3 Views