In our latest runs, I've noticed some unusual behavior - the majority of our assets are never enter into the preparing phase / unaccounted for. Basically, many assets without any upstream dependencies never get scheduled. I also see assets being "prepared" with upstream dependencies that have fail, but those dependencies jsut hang in preparing. A week ago this was operating as expected - I'm wondering if what would have caused a regression.
I experimented with setting
DefaultRunCoordinator
and setting
QueuedRunCoordinator
with and without
max_concurrent_runs
, but found no difference in behavior. I'm also using a
K8sRunLauncher
with ample CPU and RAM.
Here's an example of what i mean by not accounted for
78+1+6+88 = 173 in a phase, but there are 459 assets in the graph. Where are the other 286?
contrast that with a week ago..
289 + 38 + 121 = 448 assets