https://dagster.io/ logo
Title
a

Andrew

02/20/2023, 4:48 PM
My multipartitioned backfills are failing, and I don't understand why. Below is a small example that I am using for testing. When I manually materialize a single asset, this will run fine. But if I queue up a backfill to run multiple at the same time, it fails (error below).
@asset(
    name="state_link",
    partitions_def=foo_monthly_site_partition_def,
    code_version="0.1",
)
def scheduled_state_partition(context) -> Output[str]:
    partition_keys = context.partition_key.keys_by_dimension
    date_partition = partition_keys.get("date")
    state_partition = partition_keys.get("state")
    # create state zip link
    area_link = f"<https://www.foo.com/{state_partition}/zip-codes/>"

    return Output(
        area_link,
        metadata={"date_partition": date_partition, "state_partition": state_partition},
    )
Error:
AttributeError: 'str' object has no attribute 'keys_by_dimension'
  File "/opt/conda/envs/dagster/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 55, in op_execution_error_boundary
    yield
  File "/opt/conda/envs/dagster/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 457, in iterate_with_context
    next_output = next(iterator)
  File "/opt/conda/envs/dagster/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 121, in _coerce_solid_compute_fn_to_iterator
    result = invoke_compute_fn(
  File "/opt/conda/envs/dagster/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 115, in invoke_compute_fn
    return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
  File "/usr/app/hydraulis/collectors/foo_site/assets/placards.py", line 37, in scheduled_state_partition
    partition_keys = context.partition_key.keys_by_dimension
The concepts page on backfills (https://docs.dagster.io/concepts/partitions-schedules-sensors/backfills) is very high level and doesn't address this. The resources around multi-partitions (which I understand are in beta) are also not super well supported with examples. Can anyone help enlighten me on this? Why is the context.partition_key now producing a string instead of a partition_key? using dagster 1.1.13. EDIT: This issue was fixed in 1.1.18. Leaving post up for anyone who searches slack like I did.
c

claire

02/21/2023, 10:24 PM
Hi Andrew, yep, there was a recent regression with the
context.partition_key
field in 1.1.13 where the returned value was a string instead of a
MultiPartitionKey
. This is now fixed as of 1.1.18. If you have feedback on how to improve documentation around multipartitions, please feel free to share!