https://dagster.io/ logo
#dagster-serverless
Title
# dagster-serverless
j

Jacob Marcil

02/16/2023, 9:05 PM
Hello everyone, I’m trying to configure 2 assets that have the same MultiPartitionKey. The first asset work, but on the second one I got this error.
Copy code
ValueError: Tried to get asset partitions for an output that correponds to a partitioned asset that is not partitioned with a TimeWindowPartitionsDefinition.
Anyone know what I should do?
c

claire

02/16/2023, 10:20 PM
Hi Jacob, would you mind pasting a code snippet of how you're reaching this error so I can repro?
j

Jacob Marcil

02/16/2023, 10:22 PM
So i think I found the error, but looking at the code I think it’s not yet implemented. Does Snowflake IO Manager handle
MultiKeyPartition
and if so how can I tell him which one is static and which one is the time windows function. Right now the only configuration parameter for partition seems to be
partition_expr
and this needs to be a Time windows. Right?
Here’s a code snippet, but I don’t know how much this will help.
Copy code
@asset(
    partitions_def=dar_multi_partitions,
    op_tags={"kind": "snowflake"},
    io_manager_key="warehouse_io_manager",
    key_prefix=["dar"],
    required_resource_keys={"snowflake"},
    metadata={"partition_expr": "effectivetime"},
)
def dar_hourly_token_price(dar_hourly_token_price_s3) -> DataFrame:
    payload = json.loads(dar_hourly_token_price_s3)

    df = pd.json_normalize(payload["prices"], sep="_")

    return df
dar_hourly_token_price_s3
is partition with the same
partitions_def
.
c

claire

02/16/2023, 10:25 PM
That's right, it's not supported as of 1.1.18. But in the dagster version coming out this week (1.1.19) the snowflake IO manager will support multipartitions cc @jamie
j

jamie

02/16/2023, 10:28 PM
for multi partition support, the partition_expr will be able to take a dictionary to specify the column for each dimension of the partition. example:
Copy code
@asset(
    partitions_def=MultiPartitionsDefinition(
        {
            "date": DailyPartitionsDefinition(start_date="2023-01-01"),
            "species": StaticPartitionDefinition(
                ["Iris-setosa", "Iris-virginica", "Iris-versicolor"]
            ),
        }
    ),
    metadata={
        "partition_expr": {"date": "DATE", "species": "SPECIES"}
    },
)
There will be docs for this shortly!
j

Jacob Marcil

02/16/2023, 10:29 PM
Oh ok that’s what I though. Oh and you specifiy the type as the value. Very nice. Thank you @claire and @jamie Have a great one 🙂
🌈 1
8 Views