https://dagster.io/ logo
Title
r

Robert Kolb

04/17/2023, 6:40 PM
Hi folks, I’m having an issue with partitioning. I’ve got an asset that depends on an upstream asset which is using
DynamicPartitionsDefinition
. The downstream asset is also using dynamic partitions. When executing the downstream asset after confirming that the upstream asset has run and all partitions are populated, I get the following error:
dagster._core.errors.DagsterInvalidInvocationError: Partition range wicked_folder/basic_csv4.csv to wicked_folder/basic_csv1.csv is
        not a valid range. Nonexistent partition keys:
        ['wicked_folder/basic_csv4.csv', 'wicked_folder/basic_csv1.csv']
In the screenshot attached to this reply you can see that the partitions have run in the upstream node. The odd thing is that if I opt to run a backfill on only 1 partition, then the asset is materialized as I would expect. screenshots in the reply
this first screenshot shows that all upstream partitions have run the second screenshot shows the error message when trying to run more than 1 asset
also when i run the backfill i get a warning that says the upstream files dont exist, but can confirm that they do. I bumped my dagster/dagit version to 1.2.7 since it looked like this was an issue in an earlier version, but I'm still getting the problem
c

claire

04/18/2023, 5:08 PM
Hi Robert. Do the upstream and downstream dynamic partitions defs have the same name?