https://dagster.io/ logo
Powered by
#deployment-ecs
Title
# deployment-ecs
c

Chris Evans

08/30/2021, 3:10 PM
Hi guys, I’ve setup an ECS deploy of Dagster using terraform with the reference CloudFormation template as a guide (three seperate tasks for dagit, daemon, and user repository) . Running into this error when trying to run tasks manually. We are using templating to define the container definitions in terraform. Any thoughts on why we might be seeing this issue?
Copy code
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter containerDefinitions[0].image, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
  File "/usr/local/lib/python3.9/site-packages/dagster/core/instance/__init__.py", line 1411, in launch_run
    self._run_launcher.launch_run(LaunchRunContext(pipeline_run=run, workspace=workspace))
  File "/usr/local/lib/python3.9/site-packages/dagster_aws/ecs/launcher.py", line 97, in launch_run
    task_definition = self._task_definition(metadata, image)["family"]
  File "/usr/local/lib/python3.9/site-packages/dagster_aws/ecs/launcher.py", line 224, in _task_definition
    self.ecs.register_task_definition(**task_definition)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 677, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 725, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/usr/local/lib/python3.9/site-packages/botocore/validate.py", line 319, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
I am able to generate a similar error by running the following code which to me suggests that the image kwarg passed to 
register_task_definition
is None. https://github.com/dagster-io/dagster/blob/5db556594b050b1bb980fbae7c5725411cfbdba5/python_modules/libraries/dagster-aws/dagster_aws/ecs/launcher.py#L96
Copy code
import boto3
ecs = boto3.client("ecs", region_name='us-east-1')
ecs.register_task_definition(
        family="dagster",
        containerDefinitions=[{"image": None}],
        networkMode="bridge",
    )
j

jordan

08/30/2021, 3:30 PM
c

Chris Evans

08/30/2021, 3:36 PM
It does not set that environment variable. I’ll will add that and redeploy
I added the env variable to the user repository. That doesn’t seem to have done the trick. Any other thoughts?
j

jordan

08/30/2021, 4:38 PM
what does your workspace.yaml look like?
c

Chris Evans

08/30/2021, 4:38 PM
Copy code
load_from:
  # Each entry here corresponds to a service in the docker-compose file that exposes pipelines.
  - grpc_server:
      host: dagster-bi.dagster.local
      port: 4000
      location_name: "bi"
j

jordan

08/31/2021, 3:16 PM
That looks like I’d expect it to - I’d double check that the environment variable is actually being set in your running container.
c

Chris Evans

08/31/2021, 3:18 PM
Appreciate the help Jordan. I exec’d into the running user repository container to verify that it was set. Nonetheless, we’ve decided to try our hand deploying w/ EKS instead.
j

jordan

08/31/2021, 3:22 PM
👍 good to know - if you have any additional directions to reproduce this, I’d love to see them. The envvar isn’t a very intuitive way to make sure the repository has an image set so I imagine there are probably ways we can better source the image going forward.
3 Views
Powered by