Ross Beyer12/02/2022, 5:10 PM
chris12/02/2022, 5:30 PM
Ross Beyer12/02/2022, 5:39 PM
paradigm works just fine for monitoring a directory for new JSON files, but I was wondering if I could avoid writing a JSON file first, and have the callback function itself do something “within” the Dagster paradigm rather than writing out to disk first and having to watch the disk. How to trigger some Dagster thing from within the callback is what I’m struggling with. Could the callback function be a Dagster-decorated thing (seems like it can’t be, because the 3rd-party thing won’t pass it the right arguments)? Can the callback function emit some kind of Dagster event with the dict of data that Dagster can then treat like an op output somehow? And then Dagster could be the thing that writes out the JSON or inserts to the database and does whatever.
chris12/02/2022, 5:44 PM
Ross Beyer12/02/2022, 6:17 PM
which is a Dagster-decorated op, and then craft a callback function that basically runs
and passes the “data dict” to the
argument, and then hand that callback function to the 3rd party service to run whenever it gets a “data event” and that would then trigger a Dagster cascade of jobs (which sounds like what I want)? Sorry for being so verbose, but I’m still wrapping my head around how Dagster works.
chris12/02/2022, 6:18 PM
Ross Beyer12/02/2022, 6:24 PM
chris12/02/2022, 6:28 PM