https://dagster.io/ logo
#deployment-ecs
Title
# deployment-ecs
s

Shahab Tasharrofi

01/06/2023, 7:25 PM
Hi everyone! I’m trying to enable ECS run launcher for our company and I tried to follow @daniel’s example here and came up with the following config in
dagster.yaml
but I’m getting an error back from AWS saying:
botocore.exceptions.ParamValidationError: Parameter validation failed: Invalid type for parameter networkConfiguration.awsvpcConfiguration.assignPublicIp, value: False, type: <class 'bool'>, valid types: <class 'str'>
config in `dagster.yaml`:
Copy code
run_launcher:
  module: "dagster_aws.ecs"
  class: "EcsRunLauncher"
  config:
    include_sidecars: true
    run_task_kwargs:
      networkConfiguration:
        awsvpcConfiguration:
          assignPublicIp: false
          securityGroups:
            - sg-*****
          subnets:
            - subnet-****
            - subnet-****
A few notes: • I have checked with our DEVOPS and the values of security groups and subnets are correct. • I have tried many variations of this but was never able to get it to work. I’ll list some of those tries in a thread under this.
dagster bot responded by community 1
Error when trying it without any `run_task_kwargs`:
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Network Configuration is not valid for the given networkMode of this task definition.
Error when I tried this with
assignPublicIp: DISABLED
was:
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Network Configuration is not valid for the given networkMode of this task definition.
m

Mike Atlas

01/06/2023, 7:27 PM
have you tried
assignPublicIp: 'false'
? looks like it is asking for a string not a bool
d

daniel

01/06/2023, 7:28 PM
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html - I think it might be "DISABLED"? If you search the page for "assignPublicIp" that's what shows up
s

Shahab Tasharrofi

01/06/2023, 7:29 PM
@Mike Atlas: Not yet (since in Daniel’s example it was set to just true and not string
true
) but I’ll try that too @daniel: I tried that one before (see 2nd paragraph of first comment)
m

Mike Atlas

01/06/2023, 7:30 PM
what about just omitting
assignPublicIp
s

Shahab Tasharrofi

01/06/2023, 7:31 PM
Btw, just for context, we’re on 1.1.2 (not sure if that matters)
Will try that one too.
m

Mike Atlas

01/06/2023, 7:33 PM
Think you might need to specify
networkMode
to
awsvpc
as well
s

Shahab Tasharrofi

01/06/2023, 7:33 PM
That’s specified in the task def already
👍 1
Looking at AWS console, seems like
networkMode
is part of task def and
networkConfiguration
is part of service def.
m

Mike Atlas

01/06/2023, 7:38 PM
fwiw I have
networkMode
set to
awspvc
for both task defs (job runs) and service defs for our ecs deployment
s

Shahab Tasharrofi

01/06/2023, 7:39 PM
Ok. I’ll try that one too. Is it by any chance possible to have your
run_launcher
config?
m

Mike Atlas

01/06/2023, 7:40 PM
ours is simply
Copy code
run_launcher:
  module: dagster_aws.ecs
  class: EcsRunLauncher
  config:
    include_sidecars: true
    container_name: "runlauncher"
s

Shahab Tasharrofi

01/06/2023, 7:40 PM
Ok. thanks anyway.
👍 1
Just for posterity and in case someone had a similar issue in the future, the minimal configuration that ended up working for me under our awsvpc & fargate setup was: In daemon and dagit, add following to `dagster.yaml`:
Copy code
run_launcher:
  module: "dagster_aws.ecs"
  class: "EcsRunLauncher"
  config:
    include_sidecars: true
    run_task_kwargs:
      launchType: "FARGATE"
      networkConfiguration:
        awsvpcConfiguration:
          securityGroups:
            - sg-****
          subnets:
            - subnet-****
            - subnet-****
(of course security group and subnet have to come in as env vars and be injected) In the code repo export the
DAGSTER_CONTAINER_CONTEXT
env var as follows before
grpc
server is run:
Copy code
export DAGSTER_CONTAINER_CONTEXT='{"ecs":{"task_definition_arn":"'$TASK_DEF_ARN'","container_name":"'$CONTAINER_NAME'"}}'
(of course
TASK_DEF_ARN
should be extracted from aws and
CONTAINER_NAME
should refer to whatever your container is called)
🌈 1
daggy 3d 1
blob hearteyes 1
135 Views