I have an

@op

yielding `DynamicOutput`s that's taking 30m to yield 5600 outputs, taking about 0.5s between each "Yielded output" log message both when it started and when it was finishing. Is this expected? Running on k8s. I wouldn't mind except it doesn't seem like the outputs start running until the op finishes. The op is pretty simple:

@dataclass
class Test:
   test_id: str
   # etc

@op(out=DynamicOut(Test))
def yield_each_test(test_list: List[Test]):
    for test in test_list:
        yield DynamicOutput(value=test, mapping_key=test.test_id)

Is there a way to structure the

@op

more efficiently?

cc @alex

you would have to use a profiler to discern the exact cause of slowdown - i would guess its the event writes to the database

I filed this ticket to track the issue: https://github.com/dagster-io/dagster/issues/11475 . (Jamie and Alex, I think what you're saying is that the slowdown is likely on the Dagster framework side, it doesn't look like my op is doing something obviously slow.)