Folks we have some tasks on Dagster that get stuck on

Starting

on the interface. After some painful tshooting, I see that the stoppedReason is

Timeout waiting for network interface provisioning to complete.

which is documented as a possible flakyness on ECS Scheduler and requires a manual attempt to run again, but I don’t think

dagster-ecs

is handling this. We are using v`0.15.9.`

Hi Bianca - I believe in the latest version of dagster this will be detected and moved into a failure state, and a run-level retry can be set up to restart the job on failure

Great, thanks! - do you happen to know if I can find this change on dagster-ecs changelog?

I would actually expect this to be available in 0.15.9 if run_monitoring is enabled on your instance: https://docs.dagster.io/0.15.9/deployment/run-monitoring#run-monitoring

Just to clarify, is this with cloud or open source?

Open source!

ECSRunLauncher should be in that list - we'll get that added

Awesome! ty =D

@Hi Hi @daniel, cool feature! I am not able to activate the run retry in the dagster.yaml as it tells me that this is not supported for ECSRunLauncher. My environment runs on the following versions:

dagster==1.1.10 dagit==1.1.10 dagster-graphql==1.1.10 dagster-aws==0.17.10 dagster-postgres==0.17.10

And the dagster.yaml looks as follows (only included relevant part):

run_launcher:
  module: dagster_aws.ecs
  class: EcsRunLauncher
run_monitoring:
  enabled: true
  start_timeout_seconds: 600
  max_resume_run_attempts: 3
  poll_interval_seconds: 120

Run monitoring works when I remove the "max_resume_run_attempts" and "poll_interval_seconds". Is this not yet implemented for ECSRunLauncher or am I missing something? Thank you in advance!

hi Arnoud, it's a bit confusing but there's a separate 'run_retries' key for retries: https://docs.dagster.io/deployment/run-retries#configuration

run_retries:
  enabled: true
  max_retries: 3 # Sets a default for all jobs. 0 if not set

The

max_resume_run_attempts

is a separate more experimental thing that tries to keep the same run going and recover if the run process crashes, as opposed to run retries which retries in a new run if it fails

Hi @daniel, thanks for the quick response! The

run_retries

unfortunately retries all jobs that are classified as failed, which is not favorable in our setup (rerun of a failure will not resolve the underlying issue). Does the

max_resume_run_attempts

feature only retry the jobs that were stuck on starting, as that is currently what is missing in our setup? Thank you in advance!

We don't currently have support for only retrying runs that failed during starting unfortunately, although its a very reasonable feature request

Yes ofcourse! Where should I do this?

From here: https://github.com/dagster-io/dagster/issues/new/choose

No updates to share at this time on that particular issue unfortunately