19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
I'm having trouble getting a job to run which was ...
# dagster-plusz
Zach
05/03/2023, 3:01 AMI'm having trouble getting a job to run which was running fine a couple hours ago and no code changes have been made. The job seems to run its first step just fine. The first op fans out to a handful of downstream ops (in this case, 3 downstream ops are generated), but all the downstream ops just hang on STEP_WORKER_STARTING. To make it more interesting the jobs seem to be unable to be terminated - when I hit Terminate I see a log saying
[DagsterCloudAgent] Agent d91b05f0 received request to terminate runDynamicOutputs✅ 1
Zach
05/03/2023, 4:11 AMHmm so I can't even find the ECS task corresponding to the ARN that the CloudEcsRunLauncher spits out. It seems like it might be silently dying when the subprocesses are being launched
Zach
05/03/2023, 4:14 AMFound them as stopped tasks, even though the corresponding Dagster job thinks it's still spinning up subprocesses for the DynamicOutputs. They have generic
Essential container in task exitedZach
05/03/2023, 4:26 AMahh okay sweet I got it - looks like it must've been an OOM. just increased the memory on the job and it seems to have pushed through the subprocess creation for the dynamic ops. odd that it was completely silent though - I didn't even see any indication in the ECS logs. still not sure how to get these jobs to cancel though.
a
alex
05/03/2023, 3:26 PMnot sure how to get these jobs to cancel though.did you check the “force” toggle when terminating?
z
Zach
05/03/2023, 4:56 PMthe "force" checkbox wasn't showing up which was particularly odd
Zach
05/03/2023, 4:56 PMI should've taken a picture, I hope I'm not just misremembering and missed the checkbox in my debug hurry