Hi guys, I’ve setup an ECS deploy of Dagster usin...
# deployment-ecs
c
Hi guys, I’ve setup an ECS deploy of Dagster using terraform with the reference CloudFormation template as a guide (three seperate tasks for dagit, daemon, and user repository) . Running into this error when trying to run tasks manually. We are using templating to define the container definitions in terraform. Any thoughts on why we might be seeing this issue?
Copy code
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter containerDefinitions[0].image, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
  File "/usr/local/lib/python3.9/site-packages/dagster/core/instance/__init__.py", line 1411, in launch_run
    self._run_launcher.launch_run(LaunchRunContext(pipeline_run=run, workspace=workspace))
  File "/usr/local/lib/python3.9/site-packages/dagster_aws/ecs/launcher.py", line 97, in launch_run
    task_definition = self._task_definition(metadata, image)["family"]
  File "/usr/local/lib/python3.9/site-packages/dagster_aws/ecs/launcher.py", line 224, in _task_definition
    self.ecs.register_task_definition(**task_definition)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 677, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 725, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/usr/local/lib/python3.9/site-packages/botocore/validate.py", line 319, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
I am able to generate a similar error by running the following code which to me suggests that the image kwarg passed to
register_task_definition
is None. https://github.com/dagster-io/dagster/blob/5db556594b050b1bb980fbae7c5725411cfbdba5/python_modules/libraries/dagster-aws/dagster_aws/ecs/launcher.py#L96
Copy code
import boto3
ecs = boto3.client("ecs", region_name='us-east-1')
ecs.register_task_definition(
        family="dagster",
        containerDefinitions=[{"image": None}],
        networkMode="bridge",
    )
j
c
It does not set that environment variable. I’ll will add that and redeploy
I added the env variable to the user repository. That doesn’t seem to have done the trick. Any other thoughts?
j
what does your workspace.yaml look like?
c
Copy code
load_from:
  # Each entry here corresponds to a service in the docker-compose file that exposes pipelines.
  - grpc_server:
      host: dagster-bi.dagster.local
      port: 4000
      location_name: "bi"
j
That looks like I’d expect it to - I’d double check that the environment variable is actually being set in your running container.
c
Appreciate the help Jordan. I exec’d into the running user repository container to verify that it was set. Nonetheless, we’ve decided to try our hand deploying w/ EKS instead.
j
👍 good to know - if you have any additional directions to reproduce this, I’d love to see them. The envvar isn’t a very intuitive way to make sure the repository has an image set so I imagine there are probably ways we can better source the image going forward.