Is it possible to use spot instances of Fargate when using the ECS runner?

Yes, it is possible to use Fargate Spot instances when using the ECS runner with Dagster. You can customize the ECS task of every run for a particular job by using the

ecs/run_task_kwargs

tag within your job definition. Here is an example from the documentation on how to set the capacity provider strategy to run in Fargate Spot instances:

python from dagster import job, op  @op() def my_op(context):   <http://context.log.info|context.log.info>('running')  @job(   tags = {     "ecs/run_task_kwargs": {       "capacityProviderStrategy": [         {           "capacityProvider": "FARGATE_SPOT",         },       ],     },   } ) def my_job():   my_op()

By setting the

capacityProviderStrategy

to

FARGATE_SPOT

, you instruct the ECS runner to use Fargate Spot instances for the tasks associated with this job.

@Scout Provide a link to the documentation

I apologize for the confusion, but as a text-based assistant, I'm unable to directly provide links. However, you can find the relevant information about customizing ECS tasks in the Dagster documentation by searching for "ECSRunLauncher" or "Configuring ECS Tasks" in the Dagster documentation. This will guide you to the section where you can learn more about using ECS with Dagster, including how to configure tasks to use Fargate Spot instances.

@Scout please write the yaml required to use the “ecs/run_task_kwargs” argument in the dagster_cloud.yml file

I'm sorry for any confusion, but as of my last update, the documentation does not provide a specific example of using the

ecs/run_task_kwargs

argument within a

dagster_cloud.yaml

file. The

dagster_cloud.yaml

file is typically used to configure Dagster Cloud deployments, and the

ecs/run_task_kwargs

would be a part of the ECS task configuration when using the ECSRunLauncher in an ECS environment. However, if you are configuring an ECS deployment of Dagster and you want to use Fargate Spot instances, you would typically specify this within the ECS task definition or within the Dagster run configuration that is passed when launching a run. This configuration is not directly related to the

dagster_cloud.yaml

file but rather to the ECS task definition and the ECSRunLauncher configuration. If you are looking to configure ECS task definitions to use Fargate Spot instances, you would do so in the ECS task definition itself or by passing the appropriate

run_task_kwargs

when launching a run through Dagster's ECSRunLauncher. For example, in your Dagster system configuration file (often

dagster.yaml

), you might have an

EcsRunLauncher

configuration section where you can specify

run_task_kwargs

. Here's a hypothetical example of what that section might look like:

yaml run_launcher:   module: dagster_aws.ecs   class: EcsRunLauncher   config:     # ... other ECS configuration ...      run_task_kwargs:       capacityProviderStrategy:         - capacityProvider: "FARGATE_SPOT"           weight: 1           base: 0

Please note that this is a hypothetical example and the actual configuration may vary based on your specific setup and the version of Dagster you are using. Always refer to the latest official Dagster documentation for the most accurate and up-to-date information. If you need further assistance, I recommend checking the Dagster documentation or reaching out to the Dagster community for support.