https://dagster.io/ logo
#integration-snowflake
Title
# integration-snowflake
m

Matt Clarke

06/05/2023, 5:42 PM
In the context of self-dependant assets, what would be the way to make a time-partitioned asset dependant on all past partitions of itself? We are wanting to construct an events table, where in order to calculate events on date t, we need all previous events. I see
AllPartitionMapping
, but I think that would get confused with backfills
s

sandy

06/05/2023, 10:09 PM
There isn't currently an explicit way to do this, but if you assign
start_offset
to a large negative number, I believe you'll get this behavior.
m

Matt Clarke

06/06/2023, 12:58 AM
Makes sense, thanks :)
Something we noted with this. This asset is downstream of a partitioned observable source asset, and we're seeing
Copy code
<partitions> cannot be materialized because upstream materializations are missing.
Even when we disabled the partition mapping this was the case, so I think it may be an issue with dependency resolution for partitioned source assets.
s

sandy

06/06/2023, 10:54 PM
Where are you seeing that?
m

Matt Clarke

06/06/2023, 11:02 PM
I have a daily partitioned observable source asset with a snowflake io manager, and downstream of that I have a self-dependent asset events_table with the partition mapping described above. When I go to materialise the events_table, it warns me that some upstream materialisations are missing
If I ignore the warning and run it, it materialises fine so the warning isn’t accurate. In the case of the first partition of the events table, there are no materialisable assets upstream of it (no prior partitions of itself, and only a source asset upstream)
s

sandy

06/06/2023, 11:19 PM
Got it - that does sound like a bug in the reporting. I filed an issue to track this: https://github.com/dagster-io/dagster/issues/14662.
m

Matt Clarke

06/06/2023, 11:23 PM
Thanks :) I’ll put together a minimal example to reproduce the issue tomorrow morning