https://dagster.io/ logo
Join Slack
Powered by
If one asset is dependent on multiple assets, will...
# ask-community
a
If one asset is dependent on multiple assets, will it only get materialised when all of the dependents are materialised?
y
yes
a
Thanks!
can we make the dependency optional so that it works even if one of the dependents is materialised.. @yuhan
We have a process where we have a set of tables defined as assets and there's a view which unions the data from those tables. The view is defined as an asset as well. Now, it can happen that only one of theose tables materialise at a time but we still want the view to materialise which would process the following materialisations. How can I make this happen? Right now, the view is not materialising because all of the tables haven't materialised.
e
You could have a sensor that materializes the downstream asset when ANY of the watched assets materialize.
a
Where would I add this logic? We are using reconciliation sensor which starts to materialise the view asset I mentioned above and other assets that follow. But, right now, there are multiple table assets that are flowing into the view asset. So, view asset, is waiting for all of those to get materialised..
e
It would be a totally separate sensor (multi asset sensor is best) that watches the input assets, let's assume A -> D, and yields a RunRequest to materialize the view asset, when any of them have triggered. This is just python logic you would write into the sensor.. This shouldn't interfere with the reconciliation sensor. It's independent.
a
Ah okay.. makes sense. Thanks @Emir Karamehmetoglu!
I am seeing a strange behaviour - I have 3 assets - A, B and C. C has A and B asset as non-argument dependents and we have Asset Reconciliation Sensor which is meant to materialise C. Now, when the code is first loaded and only A has materialised, then the reconciliation sensor doesn't kick in. When B gets materialised at a later point, the reconciliation sensor kicks in and C gets materialised. From now on, even if one of A-B asset gets materialised, C gets materialised again which is what we wanted all along. The behaviour is a bit strange here. Looks like C only needed A and B to get materialise once (albeit at different times) and after that C can get materialised even after only one of the upstream assets get materialised. This is definitely a bug, right? @yuhan
Based on the description here - https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#asset-reconciliation-sensors, the asset reconciliation sensor should reconcile the assets if any of its parents are materialised. In my case, Asset A gets materialised and B hasn't been materialised yet. I was expecting C to still get materialised as A got materialised. At a later time, when B gets materialised, C can get rematerialised as one of its parents have materialised more recently than it has. The above is not happening at the moment..
o
hey! this is a reasonable confusion -- the reason that it works this way is that in most cases, it is not possible to materialize an asset before both of its parents have been materialized at least once. E.g. imagine that C was pulling in actual dataframe objects from its upstreams, this operation would fail if B did not exist the way I would phrase it is that downstream assets only get materialized when a) they are unreconciled and b) both of their parents are reconciled where "reconciled" means "exists and has pulled in the most recent changes from its parents"
a
Got it, thanks @owen
s
Hey @Abhishek Agrawal - we've been working on designs for a UI that will make it easier to understand why assets are or aren't being materialized by the reconciliation sensor. Here's a github issue with a screenshot: https://github.com/dagster-io/dagster/issues/13331 I'm curious if you have any feedback or thoughts on the best way to present this information
a
Thanks Sandy. I was away, resuming work today. Will get to this today..
257 Views
Open in Slack
PreviousNext
Powered by