03/06/2023, 9:35 PM
Hi, i have source tables (e.g. in BigQuery) outside of my control - from another team - as input for my assets. New partitions of the source tables should trigger my assets to be rebuild with a new daily partition as well. Each new daily partition of the source tables should be a parameter passed to my assets for materialization. What would be the best way to achieve this in Dagster?
combined with staleness + observations seem to be a great fit - but i believe it is not yet feature rich enough? Can SourceAssets be checked continuously like sensors? Can observable source assets trigger events with parameters (like a new business date)? If not, sensors can solve the problem but seem overly complicated, please take a look at at the attachment. Looking forward for any feedback / suggestions.


03/08/2023, 4:56 PM
Hey @kern - thanks for putting this together. Would you be interested in chatting about this? Observing source assets automatically is an area of the system we've been thinking through recently. cc @sean


03/14/2023, 8:44 AM