I am not sure this is the right place to ask. Plea...
# announcements
t
I am not sure this is the right place to ask. Please remove if this doesn't belong here. I would like to visualize the task duration. And I have no knowledge of Air Flow and I am thinking of just reading the log file and plotting it on Dash or something similar. Could anyone point me in the right direction or approach? Currently I have no clue how to save from Dagit. Thanks!!!
a
Hi Thein. So when you say task duration do you mean the duration of the pipeline execution run or the duration of a particular solid execution?
Regardless,
execute_pipeline
returns a pipeline execution result object. This object contains event metadata about your pipeline run. (https://github.com/dagster-io/dagster/blob/7292dd6be3269f0471833e6986f3084b134e101e/python_modules/dagster/dagster/core/execution/results.py#L12). Upon inspection of those step events it should become fairly easy to grab whatever metadata you want and write it to a table or pipe it out to a file somewhere but if you want more, you can check out the DagsterEvent object to learn more (https://github.com/dagster-io/dagster/blob/29dda222e31bd61b861b8bc3b4e272e111348420/python_modules/dagster/dagster/core/events/__init__.py#L146)
I would also love to learn more about your dash-boarding use case. What metrics are you looking to track?
t
Hi Abhi, thanks for the reply. This is really helpful. As of now, I just want to monitor which ML models run faster and compare between each run. It is very fundamental at this point. And another use case is to monitor how many each model run every day.
a
That makes total sense. This should be sufficient for answering those flavors of questions. If not, let us know and we can go deeper.