is there a canonical way to log metadata to the console spec dagster #ask-community

is there a canonical way to log metadata to the co...

Zach

08/25/2022, 3:45 PM

is there a canonical way to log metadata to the console (specifically clickable URL links)? I've been using

context.instance.report_engine_event()

with an

EngineEventData

object, but it seems that

EngineEventData

isn't in the public API for 1.0+, and

context.log_event()

appears to only be used for logging AssetMaterializations/Observations and ExpectationResults

Zach

08/25/2022, 3:51 PM

also, it'd be nice to get

StepExecutionContext

and

StepRunRef

in the public API for annotations when writing custom step launchers

chris

08/25/2022, 5:37 PM

Good point regarding

StepExecutionContext

StepRunRef

chris

08/25/2022, 5:38 PM

As far as logging arbitrary metadata - I don't think we have a way to do this yet. A workaround could be yielding a dummy output which contains a clickable link as metadata.

chris

08/25/2022, 5:39 PM

(Or, if you don't mind the metadata being linked to an output, just put the metadata on the relevant output)

Zach

08/25/2022, 5:40 PM

maybe this is a silly question but can you yield an output in the middle of an op? in particular I'm using this metadata to provide clickable links to logs / status pages where remote execution is taking place, in often long-running ops, so it's nice to have the link early on in the op logs

chris

08/25/2022, 5:58 PM

yep! op outputs can be yielded: https://docs.dagster.io/concepts/ops-jobs-graphs/op-events#output-objects

🌈 1

Zach

08/25/2022, 6:03 PM

thanks a bunch!

4 Views

Open in Slack

Previous Next