Brendan Couche
01/26/2023, 4:55 PMEcsRunLauncher
. I've packaged everything up in the same container (dagit
, dagster-daemon
, and pipeline code). Infrastructure is managed with Terraform and everything seems to deploy cleanly. Unfortunately, when I try to kick off a task I'm greeted with the following exception:
dagster._check.ParameterCheckError: Param "image" is not a str. Got None which is type <class 'NoneType'>.
File "/opt/venv/lib/python3.10/site-packages/dagster/_core/instance/__init__.py", line 1966, in launch_run
self.run_launcher.launch_run(LaunchRunContext(pipeline_run=run, workspace=workspace))
File "/opt/venv/lib/python3.10/site-packages/dagster_aws/ecs/launcher.py", line 330, in launch_run
run_task_kwargs = self._run_task_kwargs(run, image, container_context)
File "/opt/venv/lib/python3.10/site-packages/dagster_aws/ecs/launcher.py", line 513, in _run_task_kwargs
task_definition_config = DagsterEcsTaskDefinitionConfig.from_task_definition_dict(
File "/opt/venv/lib/python3.10/site-packages/dagster_aws/ecs/tasks.py", line 120, in from_task_definition_dict
return DagsterEcsTaskDefinitionConfig(
File "/opt/venv/lib/python3.10/site-packages/dagster_aws/ecs/tasks.py", line 53, in __new__
check.str_param(image, "image"),
File "/opt/venv/lib/python3.10/site-packages/dagster/_check/__init__.py", line 1365, in str_param
raise _param_type_mismatch_exception(obj, str, param_name, additional_message)
I'm using the default run coordinator and the following config for EcsRunLauncher
:
run_launcher:
module: dagster_aws.ecs
class: EcsRunLauncher
config:
container_name: "dagster_task"
include_sidecars: true
Any help would be much appreciated ๐runs.run_body
(DB) on one of the tasks:
"pipeline_code_origin": {
"__class__": "PipelinePythonOrigin",
"pipeline_name": "__ASSET_JOB",
"repository_origin": {
"__class__": "RepositoryPythonOrigin",
"code_pointer": {
"__class__": "PackageCodePointer",
"attribute": "emotive_repository",
"module": "data_director",
"working_directory": "/home/app/dagster_app"
},
"container_context": {},
"container_image": null,
"entry_point": ["dagster"],
"executable_path": "/opt/venv/bin/python"
}
}
Mike Atlas
01/26/2023, 5:15 PMimage
is one of the fields in an ECS Task Definition (which are kinda like kubernetes manifests)container_definitions
Brendan Couche
01/26/2023, 5:16 PMdagster-daemon
and dagster-web
)
3. I'm using the DefaultRunCoordinator
to spin tasks up immediately with the above EcsRunLauncher
config - my understanding was that this would mean inspecting the existing task definition and running a duplicate (with different entrypoint) for tasks that are spun up.Mike Atlas
01/26/2023, 5:20 PMBrendan Couche
01/26/2023, 5:26 PM{
"taskDefinitionArn": "arn:aws:ecs:us-west-2:<ACCT>:task-definition/dagster-daemon:15",
"containerDefinitions": [
{
"name": "dagster-daemon",
"image": "<ACCT>.<http://dkr.ecr.us-west-2.amazonaws.com/dagster:<COMMIT|dkr.ecr.us-west-2.amazonaws.com/dagster:<COMMIT>>",
"cpu": 0,
"portMappings": [
{
"containerPort": 8000,
"hostPort": 8000,
"protocol": "tcp"
}
],
"essential": true,
"command": [
"dagster-daemon",
"run"
],
"environment": [
...
],
"mountPoints": [],
"volumesFrom": [],
"linuxParameters": {
"initProcessEnabled": true
},
"secrets": [
...
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/dev/data-platform/dagster-daemon",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "dagster-daemon"
}
},
"healthCheck": {
"command": [
"CMD-SHELL",
"ps -eax | grep -v grep | grep dagster-daemon || exit 1"
],
"interval": 60,
"timeout": 30,
"retries": 3,
"startPeriod": 5
}
}
],
"family": "dev-dagster-daemon",
"taskRoleArn": "arn:aws:iam::<ACCT>:role/<role-name>",
"executionRoleArn": "arn:aws:iam::<ACCT>:role/<role-name>",
"networkMode": "awsvpc",
"revision": 15,
"volumes": [],
"status": "ACTIVE",
"requiresAttributes": [
{
"name": "ecs.capability.execution-role-awslogs"
},
{
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"name": "com.amazonaws.ecs.capability.task-iam-role"
},
{
"name": "ecs.capability.container-health-check"
},
{
"name": "ecs.capability.execution-role-ecr-pull"
},
{
"name": "ecs.capability.secrets.ssm.environment-variables"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"name": "ecs.capability.task-eni"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
},
{
"name": "ecs.capability.extensible-ephemeral-storage"
}
],
"placementConstraints": [],
"compatibilities": [
"EC2",
"FARGATE"
],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "4096",
"memory": "30720",
"ephemeralStorage": {
"sizeInGiB": 40
},
"runtimePlatform": {
"cpuArchitecture": "X86_64",
"operatingSystemFamily": "LINUX"
},
"registeredAt": "2023-01-26T16:20:28.900Z",
"registeredBy": "arn:aws:sts::<ACCT>:assumed-role/release/aws-go-sdk-1674750028660011218",
"tags": [
...
]
}
Mike Atlas
01/26/2023, 5:27 PMBrendan Couche
01/26/2023, 5:27 PMboto3.client("ecs").describe_task_definition(...)
isn't returning anything in the containerDefinitions
property and as a result, Dagster freaks outjohann
01/26/2023, 11:22 PMBrendan Couche
01/26/2023, 11:22 PMMitchell Hynes
01/27/2023, 2:31 AMdocker compose
ECS example. I have very little changed except my user_code
is a separate compose definition, so we can have multiple code locationsservices:
user_code:
platform: linux/amd64
build:
context: .
dockerfile: ./Dockerfile
target: user_code
image: "$REGISTRY_URL/deploy_ecs/user_code"
container_name: user_code
command: "./env-wrap.sh dagster api grpc -h 0.0.0.0 -p 4000 -f pipelines.py"
Hereโs the other one
services:
dagit:
platform: linux/amd64
build:
context: .
dockerfile: ./Dockerfile
target: dagit
image: "$REGISTRY_URL/deploy_ecs/dagit"
container_name: dagit
command: "./env-wrap.sh dagit -h 0.0.0.0 -p 3000 -w workspace.yaml"
...
daemon:
platform: linux/amd64
build:
context: .
dockerfile: ./Dockerfile
target: dagster
image: "$REGISTRY_URL/deploy_ecs/daemon"
container_name: daemon
command: "./env-wrap.sh dagster-daemon run"
Sorry for muddying up the threadBrendan Couche
01/27/2023, 3:09 AMcontainer_name
specified for the EcsRunLauncher
from what was defined in the containerDefinitions
property of the ECS task definition. I'm not trying the compose route, so I'm not sure how much help that'll be ๐