Mark Fickett
03/30/2022, 4:04 PM@op
in the pipeline started? I know Dagster has an internal event log and a GraphQL API (looks like the Python API client doesn't touch on events). Would it be crazy to get the current run ID from context, query GraphQL to get the timestamp of an earlier op start, and use that to compute/log elapsed time for some portion of the DAG? Is there a simpler way to access events from within a running job?owen
03/30/2022, 4:22 PMinstance
property, which can be used to interact with the active DagsterInstance. So for your use case you could probably do something using context.instance.all_logs(context.run_id, DagsterEventType.STEP_STARTED)
, then filter for the specific op you're interested in from thereMark Fickett
03/31/2022, 2:34 PMcontext.instance.all_logs
will work to get records of events from `@op`s run previously on different Kubernetes nodes? I've been testing it with local python multithreading, just want to make sure it'll keep working in our planned deployment environment.