https://dagster.io/ logo
#ask-ai
Title
# ask-ai
i

Ismael Rodrigues

03/01/2024, 5:20 PM
from dagster import ( asset, multi_asset, AssetOut, MultiPartitionsDefinition, StaticPartitionsDefinition, DailyPartitionsDefinition, MultiToSingleDimensionPartitionMapping, AssetIn, ) multi_partitions_def = MultiPartitionsDefinition({ "entity": StaticPartitionsDefinition(["DOG", "CAT"]), "date": DailyPartitionsDefinition(start_date="2023-12-01", timezone="America/Fortaleza") }) single_partitions_def = DailyPartitionsDefinition(start_date="2023-12-01", timezone="America/Fortaleza") @asset(partitions_def=multi_partitions_def) def my_asset_1(): # Logic for the multi-dimensional asset ... @multi_asset( outs={"my_multi_asset_2": AssetOut(), "my_multi_asset_3": AssetOut()}, ins={"my_asset_1": AssetIn(partition_mapping=MultiToSingleDimensionPartitionMapping("date"))}, partitions_def=single_partitions_def ) def my_multi_asset_2(my_asset_1): # Logic for the single-dimensional @multi_asset that depends on my_asset_1 yield Output(data, output_name=logic_to_set_name) @dbt_asset( partitions_def=single_partitions_def, all the other args ) def dbt_asset(my_multi_asset_2): # Logic for the dbt asset that depends on my_multi_asset_2 ... I have this code, but the mapping between multiple dimension keys doesn't happen to a single dimension date on the multi_asset, the auto materialization stills waiting for the main asset materialize without the dimensions its has
s

Scout

03/01/2024, 5:20 PM
Based on the documentation, when you have a multi-dimensional partitioned asset and you want to create a dependency on a single-dimensional partitioned asset, you can use the
MultiToSingleDimensionPartitionMapping
to map the multi-dimensional partitions to the single dimension that they share. In your code, you have defined
my_asset_1
with a
multi_partitions_def
which partitions by both "entity" and "date". You then have
my_multi_asset_2
which is intended to depend only on the "date" dimension of
my_asset_1
. To achieve this, you've correctly used
MultiToSingleDimensionPartitionMapping("date")
in the
AssetIn
definition for
my_multi_asset_2
. However, you've mentioned that the auto materialization is still waiting for the main asset to materialize without the dimensions it has. This could be due to a few reasons: 1. The
MultiToSingleDimensionPartitionMapping
may not be set up correctly or there might be an issue with how the dependencies are being resolved. 2. There could be a bug or limitation in the version of Dagster you are using that is preventing the expected behavior. 3. The materialization logic itself might not be triggering as expected due to other reasons not directly related to the partition mapping. To troubleshoot this issue, you can: - Ensure that the
MultiToSingleDimensionPartitionMapping
is being used correctly as per the documentation. - Check if there are any known issues or updates in the Dagster GitHub repository that might address this problem. - Review the materialization and execution logs to see if there are any clues as to why the dependencies are not being resolved as expected. - Make sure that the
my_asset_1
is being materialized correctly with the expected partition keys. If after reviewing these steps the issue persists, it might be beneficial to look for similar cases in the Dagster GitHub Issues or Discussions, or consider reaching out to the Dagster community for further assistance. Remember that the documentation provides guidance on how to use these features, but if there's a discrepancy between the expected behavior and what you're experiencing, it may require a deeper investigation into the specific details of your Dagster setup and the code you're running.