https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
p

Peter Davidson

02/27/2023, 7:46 AM
Interesting error trying to use a custom partition mapping. I'm not doing anything with dynamic partitions, and surprised the error was thrown within dagster.core instead of the partition mapping I built
Copy code
An unexpected exception was thrown. Please file an issue.
TypeError: get_upstream_partitions_for_partitions() got an unexpected keyword argument 'dynamic_partitions_store'

  File "C:\Users\peter.davidson\venv\v385\lib\site-packages\dagster\_core\execution\plan\execute_plan.py", line 265, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "C:\Users\peter.davidson\venv\v385\lib\site-packages\dagster\_core\execution\plan\execute_step.py", line 344, in core_dagster_event_sequence_for_step
    for event_or_input_value in ensure_gen(
  File "C:\Users\peter.davidson\venv\v385\lib\site-packages\dagster\_core\execution\plan\inputs.py", line 522, in load_input_object
    load_input_context = self.get_load_context(step_context, input_def)
  File "C:\Users\peter.davidson\venv\v385\lib\site-packages\dagster\_core\execution\plan\inputs.py", line 475, in get_load_context
    return step_context.for_input_manager(
  File "C:\Users\peter.davidson\venv\v385\lib\site-packages\dagster\_core\execution\context\system.py", line 614, in for_input_manager
    self.asset_partitions_subset_for_input(name)
  File "C:\Users\peter.davidson\venv\v385\lib\site-packages\dagster\_core\execution\context\system.py", line 941, in asset_partitions_subset_for_input
    return partition_mapping.get_upstream_partitions_for_partitions(
s

sandy

02/27/2023, 4:51 PM
@claire - mind taking a look at this one?
p

Peter Davidson

02/27/2023, 5:26 PM
at some point,
dynamic_partitions_store: Optional[DynamicPartitionsStore] = None,
was addet to the base PartitionMapping class. I just had to add this to the implementation I wrote to allow for this, then it's working again. Took a bit of digging though.
c

claire

02/27/2023, 5:51 PM
Hi Peter. Currently we don't support overriding `PartitionMapping`s outside of dagster as the methods may change anytime. Curious about the use case for your custom partition mapping, if it's helpful for others too you can consider opening a PR to add it to dagster and we'll maintain it thereafter.
p

Peter Davidson

02/27/2023, 5:54 PM
Yes I plan to open a PR: Use case: • analysis of change of an asset between two periods. Example: • All assets multi-partition, monthly & "scenario" (Call it x, y, z) • Asset a does a calculation on a monthly basis for several scenarios (each scenairo has a different parameter) • asset b loads asset a as at a previous period (e.g. previous year end) ◦ This is where I use a partition mapping on the input to refer to a prior period • asset c does a comparison between asset a and b ps. The disclaimer in PartitionMapping is reasonably clear, but... overriding is still useful. FYI I copied the construction from the partition mapping unit tests, which was a quick start 😛
16 Views