https://dagster.io/ logo
Join Slack
Powered by
Do someone knows how to investigate those kind of ...
# ask-community
j
Do someone knows how to investigate those kind of errors? Trying to run an asset with MultiPartitionDefinition activated and receive that If the number of static partitions is high.
With that
o
hi @Jacob Marcil! how many static partitions do you have?
cc @claire
c
Would it be possible for you to paste a code snippet of how you're using a schedule to interact with multipartitions?
j
Number of static partitions : 
53
well In this example I have 47, but I have the same error.
This was my schedule configuration
Copy code
@schedule(
    cron_schedule="0 * * * *",
    job=dar_multi_partition_job,
    default_status=DefaultScheduleStatus.RUNNING,
)
def dar_schedule():
    hour = (datetime.now() - timedelta(hours=1)).strftime("%Y-%m-%d-%H:00")
    assets = ["DA3YOMC", "DA6YYNS", "DASK8KY", "DAHBT5U", "DAYT16A", "DAFB0MN", "DAU04PV", "DA05GWG", "DAY5UHJ", "DA8UISY", "DAAD2D6", "DAYK7R2", "DAEK979", "DA2Z102", "DAXVC4R", "DA14ISM", "DAYKXMG", "DAYGTPQ", "DAMFI9C", "DAD20TB", "DAEF133", "DAUY6W1", "DAMSTS2", "DAWWDPT", "DAWDCO9", "DARMN8J", "DAAFG27", "DAWPX23", "DAVTY3L", "DAWK4FA", "DA2DANE", "DAAWP08", "DARUEIM", "DAL5A0K", "DAFV08S", "DAPG1UA", "DA803TS", "DAIYNV8", "DALYJ9J", "DA4QZR7", "DAFAYJB", "DA22JZ7", "DAPV839", "DAEMBTU", "DAN0CMH", "DAWNJ9I", "DAHWGXP"]
    run_requests = []
    for asset in assets:
        run_requests.append(dar_multi_partition_job.run_request_for_partition(partition_key=f"{asset}|{hour}", run_key=f"{asset}|{hour}"))

    return run_requests
When I use less static partition I don’t have any issue. But with all of them nothing happen and I receive the error I shared in this thread.
And This is how I configured my MultiPartition
Copy code
MultiPartitionsDefinition(
    {
        "hour": HourlyPartitionsDefinition(start_date=datetime(2022, 12, 1)),
        "asset": StaticPartitionsDefinition(
            ["DA3YOMC", "DA6YYNS", "DASK8KY", "DAHBT5U", "DAYT16A", "DAFB0MN", "DAU04PV", "DA05GWG", "DAY5UHJ", "DA8UISY", "DAAD2D6", "DAYK7R2", "DAEK979", "DA2Z102", "DAXVC4R", "DA14ISM", "DAYKXMG", "DAYGTPQ", "DAMFI9C", "DAD20TB", "DAEF133", "DAUY6W1", "DAMSTS2", "DAWWDPT", "DAWDCO9", "DARMN8J", "DAAFG27", "DAWPX23", "DAVTY3L", "DAWK4FA", "DA2DANE", "DAAWP08", "DARUEIM", "DAL5A0K", "DAFV08S", "DAPG1UA", "DA803TS", "DAIYNV8", "DALYJ9J", "DA4QZR7", "DAFAYJB", "DA22JZ7", "DAPV839", "DAEMBTU", "DAN0CMH", "DAWNJ9I", "DAHWGXP"],
        ),
    }
)
Basically the same list of elements + hourly partition.
c
I was able to reproduce this. The source of the error is that 
run_request_for_partition
iterates through all the multipartitions to find the desired partition, which becomes expensive given that you have 90K partitions. I can put out a fix for this that should improve performance, in the meantime would you mind filing an issue for this?
j
Thank you for looking into this. Created this issue https://github.com/dagster-io/dagster/issues/12441
🌈 1
2 Views
Open in Slack
PreviousNext
Powered by