https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Daniel Galea

04/03/2023, 10:13 AM
Hi everyone, I am getting this error when generating a multi-asset sensor:
"All assets must be partitioned and share the same partitions definition"
. However, I'm not sure why I get this error as all my assets have the same type of partition definition and they are all partitioned. I am using this partition type
DailyPartitionsDefinition
, the difference between some of the assets is the hour and minute offset, and the start date. Do they need to have the exact same hour and minute offset, and start date too? I am creating a new
DailyPartitionsDefinition
object per asset, do all assets need to share the same definition type object too? This is how I generate a multi-asset sensor:
@multi_asset_sensor(
monitored_assets=asset_keys,
job=downstream_job,
name=sensor_name,
default_status=DefaultSensorStatus.RUNNING,
)
def trigger_daily_asset_if_both_upstream_partitions_materialized(
context: MultiAssetSensorEvaluationContext,
):
run_requests = []
for (
partition,
materializations_by_asset,
) in context.latest_materialization_records_by_partition_and_asset().items():
if set(materializations_by_asset.keys()) == set(context.asset_keys):
run_requests.append(downstream_job.run_request_for_partition(partition))
for asset_key, materialization in materializations_by_asset.items():
context.advance_cursor({asset_key: materialization})
return run_requests
I am getting the error when running this line of code:
context.latest_materialization_records_by_partition_and_asset().items()
I updated the start date for all of my assets so that they match and now the sensor is generated. This seems like a bug to me because I'd expect that all assets should have the latest partition in common at a minimum but not having to have all partitions for previous dates too. Example: I create asset A on 2023-03-04 and then later on introduce asset B on 2023-03-26. They have different start dates. Then I create a job on 2023-04-01 which will materialize asset C and is dependent on asset A and B, but I only want asset C to run from 2023-03-31 onward, so I only need the last partition to be available for both Asset A and B. This should work, no? What is understood by latest materialization records?
s

sandy

04/03/2023, 4:11 PM
Hey @Daniel Galea - you're right that this is a current limitation. Here's an issue where we're tracking making this more flexible: https://github.com/dagster-io/dagster/issues/12298
d

Daniel Galea

04/11/2023, 12:24 PM
Hey Sandy, thanks for sharing! I'll keep an eye on it 🙂
3 Views