https://dagster.io/ logo
#ask-community
Title
# ask-community
r

Rahul Dave

03/01/2023, 9:20 PM
Can one use a dagster sensor to react to an arbitrary dagster event. My guess is that since asset materializations can be reacted to in sensors ( or for that matter job success or job failure can trigger a new job using a sensor), this is how dagster does it under the hood. But the docs say that user code shoukd not emit events….
This would allow arbitrary processes to be spun off. My thinking is that as my machine learning models train WITHIN a dagster job( like for training different model types in sklearn in a loop where each iteration runs an op) I would like to fire off metric crunching jobs that react to an output event which is unconnected to any downstream input. I realize this could be done by doing the metric compute on a downstream op rather than a downstream job, or by using asset materialization sensors, but I am not using assets, just ops…
s

sandy

03/01/2023, 10:26 PM
You can use
context.instance
inside the sensor to run methods on
DagsterInstance
that fetch data from the Dagster event log Dagster doesn't have arbitrary "events" though - all the events have a type and meaning - so the event would probably need to be an
AssetMaterialization
or something
r

Rahul Dave

03/01/2023, 10:32 PM
Aha! So then an AssetMayerialization is a veneer over an output event? I guess since I don’t use assets I can just tail the event log for an appropriate op output event and use this in a non-standard way ( by which I mean as a not for consumption on input). Of course this means that this usecase is not a triggered sensor, but rather a sensor I could create with the original job using the python api…
Out of curiosity, is there a reason why creating and emitting events is not part of the public api? Is it for not wanting to support a generic event queue mechanism?
s

sandy

03/02/2023, 4:14 PM
So then an AssetMayerialization is a veneer over an output event?
Step output events indicate "this op returned a value". Asset materialization events indicate "some value was placed in persistent storage".
Out of curiosity, is there a reason why creating and emitting events is not part of the public api? Is it for not wanting to support a generic event queue mechanism?
Creating and emitting certain kinds of events (like AssetMaterializations and ExpectationResults) is part of the public API. For generic events, it's something we've lightly discussed but haven't developed a strong stance on.
r

Rahul Dave

03/02/2023, 7:05 PM
Got it, Thanks! My understanding increases daily!!!
2 Views