Hello everyone I m trying to configure 2 assets that have th dagster #dagster-serverless

Hello everyone, I’m trying to configure 2 assets ...

Jacob Marcil

02/16/2023, 9:05 PM

Hello everyone, I’m trying to configure 2 assets that have the same MultiPartitionKey. The first asset work, but on the second one I got this error.

Copy code

ValueError: Tried to get asset partitions for an output that correponds to a partitioned asset that is not partitioned with a TimeWindowPartitionsDefinition.

Anyone know what I should do?

claire

02/16/2023, 10:20 PM

Hi Jacob, would you mind pasting a code snippet of how you're reaching this error so I can repro?

Jacob Marcil

02/16/2023, 10:22 PM

So i think I found the error, but looking at the code I think it’s not yet implemented. Does Snowflake IO Manager handle

MultiKeyPartition

and if so how can I tell him which one is static and which one is the time windows function. Right now the only configuration parameter for partition seems to be

partition_expr

and this needs to be a Time windows. Right?

Jacob Marcil

02/16/2023, 10:25 PM

Here’s a code snippet, but I don’t know how much this will help.

Copy code

@asset(
    partitions_def=dar_multi_partitions,
    op_tags={"kind": "snowflake"},
    io_manager_key="warehouse_io_manager",
    key_prefix=["dar"],
    required_resource_keys={"snowflake"},
    metadata={"partition_expr": "effectivetime"},
)
def dar_hourly_token_price(dar_hourly_token_price_s3) -> DataFrame:
    payload = json.loads(dar_hourly_token_price_s3)

    df = pd.json_normalize(payload["prices"], sep="_")

    return df

dar_hourly_token_price_s3

is partition with the same

partitions_def

claire

02/16/2023, 10:25 PM

That's right, it's not supported as of 1.1.18. But in the dagster version coming out this week (1.1.19) the snowflake IO manager will support multipartitions cc @jamie

jamie

02/16/2023, 10:28 PM

for multi partition support, the partition_expr will be able to take a dictionary to specify the column for each dimension of the partition. example:

Copy code

@asset(
    partitions_def=MultiPartitionsDefinition(
        {
            "date": DailyPartitionsDefinition(start_date="2023-01-01"),
            "species": StaticPartitionDefinition(
                ["Iris-setosa", "Iris-virginica", "Iris-versicolor"]
            ),
        }
    ),
    metadata={
        "partition_expr": {"date": "DATE", "species": "SPECIES"}
    },
)

There will be docs for this shortly!

Jacob Marcil

02/16/2023, 10:29 PM

Oh ok that’s what I though. Oh and you specifiy the type as the value. Very nice. Thank you @claire and @jamie Have a great one 🙂

🌈 1

8 Views

Open in Slack

Previous Next