Hi everyone, quick question. I have an asset partitioned on the hour. Its purpose is to capture, every hour, the current state of an external system. This means that at any point in time, it is only possible to correctly materialise the latest partition. It is not possible to back-fill. How do I configure this in Dagster to prevent users from being able to back-fill from the Dagit web UI?
06/15/2023, 9:26 AM
I had this same use-case. I solved it by:
• Using an unpartitioned asset - I couldn't see a need for partitions in this case
• Writing a custom IO Manager to append to my database only. It skips the delete portion of the delete-write cycle so is not idempotent. I used a config param in the IO manager to drive the append-only logic.
• Schedule the asset to materialise on a schedule