19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi team, in an asset sensor with dynamic partition...
# ask-communityy
YH
07/18/2023, 2:45 AMHi team, in an asset sensor with dynamic partitions, I receive an error
After dynamic partitions requests are applied, it does not exist in the set of valid partition keys.YH
07/18/2023, 2:46 AMthere was an issue with the upstream and we had to do a resync of files, meaning that previous partitions were deleted
YH
07/18/2023, 6:18 AMFWIW, i tried the following
to work around this:
Copy code
clear_partitions = os.environ.get("CLEAR_PARTITIONS")
if clear_partitions == "True":
dynamic_partitions = context.instance.get_dynamic_partitions(partitions_def.name)
<http://context.log.info|context.log.info>(f"Clearing {len(dynamic_partitions)} partitions")
for partition in dynamic_partitions:
context.instance.delete_dynamic_partition(partitions_def.name, partition)
return SkipReason(f"Skipping sensor run as partitions have been cleared")n
nq8tc3y6
07/18/2023, 6:49 AMIs your problem linked to this question ?
https://dagster.slack.com/archives/C01U954MEER/p1689633766835739
y
YH
07/18/2023, 7:19 AMsimilar fashion, my issue is due to the fact that partitions are no longer available (deleted s3 keys which are used as partition keys). so my asset_sensor was choking on an error. needed to find a way to remove all the partitions. seems like wipe assets could be cleaner...
c
claire
07/18/2023, 7:13 PMHi YH. Your workaround in the asset sensor makes sense. You could optionally have an op that deletes all of the partitions in the dynamic partitions def.
I agree that it would be nice to enable deleting dynamic partitions in the UI. We have an issue for tracking here: https://github.com/dagster-io/dagster/issues/14026
Since dynamic partitions are a definition object that can be applied to many assets, I think it makes sense for wiping assets and wiping the dynamic partitions def to be independent actions.
y
YH
07/19/2023, 12:59 AMhi @claire thanks for the suggestion to have an op instead. that would be a cleaner workaround
🌈 1
YH
07/19/2023, 12:59 AMhave upvoted the the issue 🙂
2 Views