Jonathan Wears01/25/2023, 6:28 AM
prha01/26/2023, 12:50 AM
. 3. Compute logs. These are the stdout / stderr that the code emits, including from libraries like pyspark that might be not be captured in the python layer. This often includes the text output from 1 and 2. We read from the event log to present certain views in Dagit, historically: A. The run view, where all the events for a particular run is shown B. Asset materialization views, where the materialization events for a particular asset is shown. C. Retries of runs, reads from the event log to see if particular steps have succeeded/failed, in order to determine which steps should be executed in the retry. D. Step duration stats, to determine the history of step durations for a particular run. Azure blob storage would keep the history of log type 3. From the event log table, it should be safe to delete log. type 2 (would have
value of null), but it would affect the appearance of those events in scenario A. Deleting log type 1 would affect scenarios A/B/C/D, and so is much more complicated to do.
Jonathan Wears01/26/2023, 6:05 AM