https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
p

Prratek Ramchandani

05/02/2023, 1:00 PM
hey, as we refactor more of our dagster code to use declarative scheduling patterns i’ve been using observable source assets a fair bit and have some feedback. previously, if we had tables in our data warehouse that were populated by some process outside of dagster, we'd create custom sensors to check when those tables were updated and kick off downstream jobs. instead we're now representing them as observable source assets, which seems like the recommended practice: • with sensors, we could check multiple tables in each evaluation, which meant that if the compute required was an
INFORMATION_SCHEMA
query to check when those tables were updated, we could check multiple tables in one query and yield any run requests needed. there isn't a way to do this with observation functions and it feels like we're missing some notion of an observable multi source asset? • observation functions being run as jobs feels a little unintuitive - i'm using them to tell me if the asset in question has changed and think of it as a background process that evaluates every few seconds along with typical sensor evaluations i like the benefit of then being able to use auto materializations for downstream assets but would like the process of "observing" a source asset to feel more lightweight
d

Dagster Jarred

05/02/2023, 3:45 PM
cc @Sean Lopp who has brought feedback related to this
s

Sean Lopp

05/02/2023, 3:53 PM
@Prratek Ramchandani your experience matches mine, and we have an open issue to add observable multi-source assets. In the interim, this example shows how one job can check for multiple source asset updates: https://github.com/slopp/event_driven_dbt_assets @sandy I think Prratek phrases the idea that observations should be allowed in sensors even better than I did!
👀 1
s

sandy

05/02/2023, 4:55 PM
cc @sean
hey @Prratek Ramchandani - what we have in mind here is something like this:
Copy code
@observable_source_asset(
    auto_observe_frequency_minutes=60
)
def raw_data():
    ...
then, similar to with `AutoMaterializePolicy`s, the daemon would automatically kick off runs to observe the asset would this work for you?
p

Prratek Ramchandani

05/02/2023, 9:06 PM
@sandy yeah that combined with a multi-asset option would be perfect
s

sandy

05/04/2023, 10:45 PM