Pierre Cadman
03/07/2024, 1:37 PMScout
03/07/2024, 1:37 PMPartitionMapping
in Dagster is used to define the relationship between the partitions of different assets. It can be used to map partitions of an upstream asset to partitions of a downstream asset. While the documentation does not provide a direct example of creating a custom PartitionMapping
, it does mention the concept in the context of issues and discussions related to Dagster's integration with dbt. Here's an example derived from the discussions in the Dagster community, which shows how to create a custom PartitionMapping
for dbt assets: python from typing import Any, Mapping, Optional from dagster import PartitionMapping, TimeWindowPartitionMapping from dagster_dbt import DagsterDbtTranslator, dbt_assets class CustomDagsterDbtTranslator(DagsterDbtTranslator): @classmethod def get_partition_mapping( cls, dbt_resource_props: Mapping[str, Any], dbt_parent_resource_props: Mapping[str, Any], ) -> Optional[PartitionMapping]: custom_partition_mappings = { ( "model.jaffle_shop.customers", "model.jaffle_shop.customers", ): TimeWindowPartitionMapping(start_offset=-1), # ... add more custom mappings as needed } dbt_unique_id = dbt_resource_props["unique_id"] dbt_parent_unique_id = dbt_parent_resource_props["unique_id"] dbt_dependency = (dbt_unique_id, dbt_parent_unique_id) return custom_partition_mappings.get(dbt_dependency) @dbt_assets(manifest=..., dagster_dbt_translator=CustomDagsterDbtTranslator()) def my_custom_dbt_assets(): ...
In this example, CustomDagsterDbtTranslator
is a subclass of DagsterDbtTranslator
that overrides the get_partition_mapping
method. This method defines a custom mapping between dbt assets using their unique identifiers. The TimeWindowPartitionMapping
is used to specify the relationship between the partitions, with a start_offset
indicating the relative position of the partitions. Please note that this example is based on a discussion from the Dagster community and is not a direct code example from the official documentation. For specific use cases and further details on implementing custom PartitionMapping
, you should refer to the official Dagster documentation or seek guidance from the Dagster community.