Hey Team, I am having a job where i am using the same op definition (for great expectations) to create 2 ops using an alias. The op yields 3 metadata values. When i run it, in Dagit, it see the output of both events, but when i read the event logs from a sensor, only one op's output is present and that too with only the first metadata value output. Can anyone shed some light on this?

hi @Vignesh M could you share the code for this sensor?

Hi @owen, PFA the code. The ops marked in the image attached here are the ops in question. Each of them have 3 ebent metadata entries, but only the first entry of the op on the left is present in event logs when we see it in sensor.

thanks for sharing -- to be clear, when you say that only one op's output is present in the event logs when you read from a sensor, how are you determining that? Are you directly reading the eventmetadatalog.txt file, or does the event_logs_metadata value not contain what you're expecting? If it's the second thing, do you mind sharing what's happening in

get_event_logs_metadata()

? The command

instance.get_all_logs(run_id)

should basically return exactly the list of events that you see in Dagit, so it would definitely be strange if there was a difference between the events seen there and in dagit.

i am writing the output of instance.get_all_logs to a file just to see if everything gets captured using the below lines

event_logs = context.instance.all_logs(run_id)
with open("D:\eventmetadatalog.txt", "w") as f:
    for x in event_logs:
        f.writelines("{0}\n".format(x))

But the output doesnt have logs for the events mentioned. Attaching the output of the Ops in dagit UI. The Metadata "Expectation Results" highlighted in red in the screenshot is the only one that gets printed. Rest 5 entries are not available.

hi @Vignesh M -- I can't think of a great reason why this would be happening. What other events exist in that file? I'd expect things like STEP_STARTED, STEP_SUCCESS, OUTPUT_HANDLED, stuff like that. Are those missing as well?

Just found out one thing: I modified the flow to look like this. Now the branch on right is not showing in event logs captured by sensor. Same case as previous scenario. Also, anything past that op are not getting captured too.

Some more things to help debug: What run status is this sensor listening for? (succeeded? failed?) In these runs, are all steps completing? You also mention that you expect the op to yield three metadata values, and it's not clear to me from the screenshot that this is not the case in the data you get from the sensor (the metadata_entries field you highlight is a list, which I would expect to have three MetadataEntry objects in it, but the first entry is very long so we can't see the end of the list)