Erik Alfthan
04/25/2023, 10:35 AM"metadata": { "partition_expr": {"<partition_column_name_1>": "anystring", "partition_column_2": "anystring"}
for for a multipartitioned snowflake io asset. It works with any dictionary, but the SQL produced is weird.
SELECT * FROM <tablename> WHERE
None in ('<partitionkey-1>') AND
None in ('<partitionkey-2>')
"metadata": {
"partition_expr": {
"<partition_column_name_1>": "<partition_column_name_1>",
"<partition_column_2>": "<partition_column_2>"
}
}
Tim Castillo
04/25/2023, 5:26 PMErik Alfthan
04/25/2023, 6:21 PMSELECT * FROM <tablename> WHERE
entity_id in ('a') AND
year_id in ('1')
but instead I get the above mentioned with None instead of column names and I can see in the snowflake query log that a query failed (since a while back, failed queries are no longer visible in the log on Snowflake unless you are accountadmin, bummer)
I can follow this up as "community service" but as I mentioned, I think I made a mistake. Each of my entities has different list of values for the year ids, so I can't use the combinatorial result I assume the multi-partition would create. I.e. if entity a has years [1,2,3], b has [1,2] and c just [1], this would not produce the correct result, because the multi partition would create all partition combinations from a|1 => c|3.Tim Castillo
04/25/2023, 6:43 PMErik Alfthan
04/26/2023, 7:23 AMlatest_materialization_event = context.instance.get_latest_materialization_events(
[context.asset_key_for_output()]
).get(context.asset_key_for_output())
if latest_materialization_event:
materialization = (
latest_materialization_event.dagster_event.event_specific_data.materialization
)
metadata = {entry.label: entry.entry_data for entry in materialization.metadata_entries}
lastmodified = metadata["lastmodified"].value
else:
lastmodified = None