Hi folks I m having an issue with partitioning I ve got an a dagster #ask-community

Hi folks, I’m having an issue with partitioning. ...

Robert Kolb

04/17/2023, 6:40 PM

Hi folks, I’m having an issue with partitioning. I’ve got an asset that depends on an upstream asset which is using

DynamicPartitionsDefinition

. The downstream asset is also using dynamic partitions. When executing the downstream asset after confirming that the upstream asset has run and all partitions are populated, I get the following error:

Copy code

dagster._core.errors.DagsterInvalidInvocationError: Partition range wicked_folder/basic_csv4.csv to wicked_folder/basic_csv1.csv is
        not a valid range. Nonexistent partition keys:
        ['wicked_folder/basic_csv4.csv', 'wicked_folder/basic_csv1.csv']

In the screenshot attached to this reply you can see that the partitions have run in the upstream node. The odd thing is that if I opt to run a backfill on only 1 partition, then the asset is materialized as I would expect. screenshots in the reply

Robert Kolb

04/17/2023, 6:43 PM

this first screenshot shows that all upstream partitions have run the second screenshot shows the error message when trying to run more than 1 asset

Robert Kolb

04/17/2023, 6:46 PM

also when i run the backfill i get a warning that says the upstream files dont exist, but can confirm that they do. I bumped my dagster/dagit version to 1.2.7 since it looked like this was an issue in an earlier version, but I'm still getting the problem

claire

04/18/2023, 5:08 PM

Hi Robert. Do the upstream and downstream dynamic partitions defs have the same name?

Kate Ross

06/12/2023, 12:54 PM

@Robert Kolb did you ever resolve this? I’m having the same issue

9 Views

Open in Slack

Previous Next