I am having an issue with Auto-materialize in 1.3.5 which produces an error that doesn't make any sense. I think the issue is that I have two partitioned sensors and the partitioned assets downstream aren't auto-materializing because there seems to be cross pollination between their keys. It appears to me that they are trying to resolve each others partition key from the wrong range of partitions so I get an error like this because this key is not in the other sensors partition range: dagster._core.errors.DagsterInvalidInvocationError: Partition range file.csv to file.csv is not a valid range. Nonexistent partition keys: ['file.csv'] File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/daemon.py", line 222, in core_loop yield from self.run_iteration(workspace_process_context) File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/asset_daemon.py", line 78, in run_iteration run_tags=None, File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/asset_reconciliation_sensor.py", line 1070, in reconcile conditions_by_asset_partition_for_freshness=conditions_by_asset_partition_for_freshness, File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/asset_reconciliation_sensor.py", line 802, in determine_asset_partitions_to_auto_materialize evaluation_time, File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/asset_graph.py", line 543, in bfs_filter_asset_partitions candidate.partition_key, File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/asset_graph.py", line 218, in get_children_partitions current_time, File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/asset_graph.py", line 270, in get_child_partition_keys_of_parent current_time=current_time, File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/partition_mapping.py", line 164, in get_downstream_partitions_for_partitions for upstream_key_range in downstream_key_ranges File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/partition.py", line 961, in with_partition_keys self._subset | set(partition_keys), File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/partition_mapping.py", line 166, in <genexpr> upstream_key_range, dynamic_partitions_store=dynamic_partitions_store File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/partition.py", line 189, in get_partition_keys_in_range {list(key for key in keys_exist if keys_exist[key] is False)}""" Is this a know issue that is being resolved or is this operator error?

hi @Chris Nogradi! what are the partitions definitions for the two assets? and what is the partition mapping between them?

Hello @owen, your question got me thinking - the issue is probably that there is an asset that depends on both sensors upstream which produce a different set of partitions. So I am guessing that my problem then is that I need to create a partition mapper that deals with the intersection of the two, right?

yep! by default, an IdentityPartitionMapping is assumed (partition

A

maps to partition

A

), but if you have different partitions definitions upstream / downstream then this would hit that error

Wow, thanks! Operator error indeed!

hm I don't think I fully understand the mapping that you're trying to construct -- it sounds like you have something like: one asset C which depends on two different assets, A and B. Which assets here share a partitions definition and how would you ideally like to set up the dependencies?

Sorry @owen about the confusion. Yes, C depends on A and B. C shares a partition def with A but C has its own since assets can only have one partition def. So, being ignorant on how to make this work, I tried putting a mapping on the AssetIn() for the B input parameter to properly map that input key from the upstream B key based on the C partition dep. But that does not seem to be supported with the mapping options available. I am probably doing this all wrong and don't mind you telling me so. ;)