The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

So....conditional asset materialisations (only materialise if the new asset would be "different" to the old one). Is there a way I'm missing?

see <https://dagster.slack.com/archives/C01U954MEER/p1652905382653489?thread_ts=1652876973.832529&amp;cid=C01U954MEER>

if you mean dynamic partitions discussion is ongoing <https://github.com/dagster-io/dagster/discussions/6913>

also see my points yesterday for some problems I face at the current stage. (without full support of dagster for dynamic partitions)

Ah! Nice one tyvm! Will have a fiddle with this today and see how I get on

So the `multi_asset` "hack" works nicely except the condition I wish to evaluate is whether this new materialisation would be different to the current state of the asset and I...don't think I can access that here?

I mean, in this case it's a file-based asset so I can see it in the relevant place the io-manager put it but it feels a bit dirty to just read it from there?

no I think the IO manager is the wrong place. The logic if it is new should go into the definition of the asset

Yeah the logic is in the asset but the logic I need relies on the *previously materialised version* of that asset which I don't think I can (or maybe just don't know how) access from within the asset definition

no you do can access that by retrieving the ASSET_MATERIALIZATION Event for your specific asset key.

I.e. you could always retrieve the latest one easily

It gets more complicated when a specific one should be providable in edge cases i..e when backfilling data

hm ok - any guide/docs on this around? No backfilling edge cases here