The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

For the observable source assets <https://docs.dagster.io/concepts/assets/asset-observations#observable-source-assets> how can I combine an automatic observation like a sensor with them? The documentation seems to only show the refresh sources button. In particular - how does this go well with DBT?

<https://dagster.slack.com/archives/C01U954MEER/p1678138555114909>

<@U01PP2ED4ES> adding support for observable source assets to the reconciliation sensor is planned. <@U011CET83FG> would have the best details on the timeline.

Assuming your dbt models managed by dagster are downstream of an observable source asset, an updated “observation” would lead to a dbt run

<@U01PP2ED4ES> - as of today's release, you can now schedule observations of source assets: <https://docs.dagster.io/concepts/assets/asset-observations#observable-source-assets>

What is the reason for: *NOTE*: Currently, source asset observations cannot be run as part of a standard asset job that materializes assets. The `selection` argument to `define_asset_job` must target only observable source assets-- an error will be thrown if a mix of regular assets and observable source assets is selected. ?

<@U01PP2ED4ES> this is a limitation as the feature is being fully implemented.

Today you can use two schedules
(1) a schedule for observations that marks downstreams stale
(2) a schedule for assets that launches RunRequests with `stale_assets_only`

Soon we will support combining observations and downstream materializations more directly via schedules or a reconciliation sensor