https://dagster.io/ logo
Join Slack
Powered by
I am having trouble using the `ECSRunLauncher` alo...
# deployment-ecs
j
I am having trouble using the 
ECSRunLauncher
along with a specified 
task_definition
. When I execute a pipeline from dagit, ECS correctly starts up the runner task definition, but the container running is not overridden with the user code docker image. Is there some configuration I am missing? I will add config notes in thread. The user code image has 
DOCKER_CURRENT_IMAGE
 
DAGSTER_CURRENT_IMAGE
defined. The 
task_defintion
has a single container defined with image which is the image running dagster-daemon/dagit. We define the 
task_definition
as an environment variable passed to dagit and dagster-daemon.
d
Ah, it's DAGSTER_CURRENT_IMAGE not DOCKER_CURRENT_IMAGE
j
My mistake, 
DAGSTER_CURRENT_IMAGE
is defined, not 
DOCKER_CURRENT_IMAGE
Log of pipeline execution params:
Copy code
INF | 2022-02-14 | 21:24:25.330836 | Begin program execution of [dagster,api,execute_run,
{
    "__class__": "ExecuteRunArgs",
    "instance_ref": {
      "__class__": "InstanceRef",
      "compute_logs_data": {
        "__class__": "ConfigurableClassData",
        "class_name": "LocalComputeLogManager",
        "config_yaml": "base_dir: /opt/dagster/dagster_home/storage\n",
        "module_name": "dagster.core.storage.local_compute_log_manager"
      },
      "custom_instance_class_data": null,
      "event_storage_data": {
        "__class__": "ConfigurableClassData",
        "class_name": "PostgresEventLogStorage",
        "config_yaml": "postgres_db:\n db_name:\n env: DAGSTER_POSTGRES_DB\n hostname:\n env: DAGSTER_POSTGRES_HOST\n password:\n env: DAGSTER_POSTGRES_PASSWORD\n port: 5432\n username:\n env: DAGSTER_POSTGRES_USER\n",
        "module_name": "dagster_postgres.event_log"
      },
      "local_artifact_storage_data": {
        "__class__": "ConfigurableClassData",
        "class_name": "LocalArtifactStorage",
        "config_yaml": "base_dir: /opt/dagster/dagster_home/\n",
        "module_name": "dagster.core.storage.root"
      },
      "run_coordinator_data": {
        "__class__": "ConfigurableClassData",
        "class_name": "DefaultRunCoordinator",
        "config_yaml": "{}\n",
        "module_name": "dagster.core.run_coordinator"
      },
      "run_launcher_data": {
        "__class__": "ConfigurableClassData",
        "class_name": "EcsRunLauncher",
        "config_yaml": "container_name: runner\ntask_definition: arn:aws:ecs:us-east-1:123456789:task-definition/team-dagster-runner:9\n",
        "module_name": "dagster_aws.ecs"
      },
      "run_storage_data": {
        "__class__": "ConfigurableClassData",
        "class_name": "PostgresRunStorage",
        "config_yaml": "postgres_db:\n db_name:\n env: DAGSTER_POSTGRES_DB\n hostname:\n env: DAGSTER_POSTGRES_HOST\n password:\n env: DAGSTER_POSTGRES_PASSWORD\n port: 5432\n username:\n env: DAGSTER_POSTGRES_USER\n",
        "module_name": "dagster_postgres.run_storage"
      },
      "schedule_storage_data": {
        "__class__": "ConfigurableClassData",
        "class_name": "PostgresScheduleStorage",
        "config_yaml": "postgres_db:\n db_name:\n env: DAGSTER_POSTGRES_DB\n hostname:\n env: DAGSTER_POSTGRES_HOST\n password:\n env: DAGSTER_POSTGRES_PASSWORD\n port: 5432\n username:\n env: DAGSTER_POSTGRES_USER\n",
        "module_name": "dagster_postgres.schedule_storage"
      },
      "scheduler_data": {
        "__class__": "ConfigurableClassData",
        "class_name": "DagsterDaemonScheduler",
        "config_yaml": "{}\n",
        "module_name": "dagster.core.scheduler"
      },
      "settings": {}
    },
    "pipeline_origin": {
      "__class__": "PipelinePythonOrigin",
      "pipeline_name": "run_load",
      "repository_origin": {
        "__class__": "RepositoryPythonOrigin",
        "code_pointer": {
          "__class__": "FileCodePointer",
          "fn_name": "repo",
          "python_file": "syncing_pipeline.py",
          "working_directory": "/opt/dagster/dagster_home"
        },
        "container_image": "docker.internal/team/staging:latest",
        "entry_point": [
          "dagster"
        ],
        "executable_path": "/usr/local/bin/python"
      }
    },
    "pipeline_run_id": "1234",
    "set_exit_code_on_failure": null
  }]
User-Code Task Definition:
Copy code
[
  {
    "name": "staging",
    "image": "docker.private/team/data-loader:latest",
    "essential": true,
    "portMappings": [
      {
        "containerPort": 8888,
        "hostPort": 8888
      }
    ],
    "environment": [
      {"name": "HTTP_PROXY", "value": "${PROXY_URL}"},
      {"name": "HTTPS_PROXY", "value": "${PROXY_URL}"},
      {"name": "NO_PROXY", "value": "${NO_PROXY}"},
      {"name": "DAGSTER_CURRENT_IMAGE", "value": "docker.private/team/data-loader:latest"},
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-region": "us-east-1",
        "awslogs-group": "${LOG_GROUP}",
        "awslogs-stream-prefix": "staging"
      }
    }
  }
]
Runner Task Definition:
Copy code
[
    {
      "entryPoint": [
        "entrypoint"
      ],
      "portMappings": [],
      "command": null,
      "environment": [],
      "image": "docker.private/team/dagster:latest",
      "essential": true
    }
]
d
And the image that it is using instead (looks like docker.internal/team/staging:latest) is the same image that's being used by the daemon/dagit?
j
no, staging is the user-code container. The 
team/dagster
is the daemon/dagit image
d
If you go to the Workspace tab in dagit, does it show the correct image under the location that you're launching the run from?
j
Yes
d
Got it - when you say "the container running is not overridden with the user code docker image." - what's your evidence that that's the case?
Oh I see, the Runner Task Definition
j
The evidence that supports container not getting overridden is that the pipeline errors out with 
"message": "Could not load pipeline definition."
combined with ECS showing the running task's container image still being the dagit/daemon image
d
Yeah, you're right - sorry, took me a bit to get there. I'm checking to see if this is expected behavior if you supply a specific task definition / even if it is, what it would take to override the DAGSTER_CURRENT_IMAGE and register a new task definition
are you hoping to use the same fixed task definition with several different locations, each supplying a different image?
j
I figured that the task definition would stay the same for different repositories and that the 
DAGSTER_CURRENT_IMAGE
defined on the repository would override the runner's task definition image
Would using the default ECSRunLauncher without a task-definition specified work in the way I described?
d
Yeah, that will definitely incorporate DAGSTER_CURRENT_IMAGE in the way you're expecting.
j
Ok, now that the new update of dagster allows you to prevent pulling secrets from AWS secret manager, we can try that method, as that was why I defined a runner task in the first place
Just curious, what was the intended purpose of being able to specify a task definition?
d
I think it was so that you could run in a custom environment that's different from the system components - but I would also expect it to override the image using DAGSTER_CURRENT_IMAGE even in that case, so we may be able to add that
j
I figure that was the impetus because it would allow you to restrict IAM permissions and security groups
👍 1
8 Views
Open in Slack
PreviousNext
Powered by