👋 Two new experimental changes dropped in yesterday’s 0.11.15 release that this channel might be interested in: The first is an ECS reference deployment that includes a step-by-step guide for deploying to ECS using docker-compose. We know docker-compose isn’t the only way to manage your infrastructure (Terraform, CloudFormation, AWS Copilot, etc.) and we also know that docker-compose’s default ECS configuration isn’t the only valid way to configure ECS, but we chose it because of its approachability and similarity to our existing deployment guides. Even if you plan to vary from what it provides by default, it might be useful to deploy the example and examine the resources it creates for inspiration on how to customize your own deployment: https://docs.dagster.io/0.11.15/deployment/guides/aws#example Additionally, we’ve created an experimental ECS Run Launcher. It too is initially limited in customizability and is specifically designed to work with the reference deployment, but we plan to extend it over the next few releases as we get more feedback. If you follow the reference deployment directions, it’ll already be included in your deployment. Otherwise, you can check out the source code here: https://github.com/dagster-io/dagster/blob/0.11.15/python_modules/libraries/dagster-aws/dagster_aws/ecs/launcher.py These are both experimental features and we’d love to get your feedback and contributions so we can continue to grow them!

Awesome work! Will have a read at somepoint this week.

I’m really excited to see this, btw. I was just wondering if there is a way to provide the

env_var

config in the

dagster.yaml

. I get an error every time I try.

What are you trying to configure and what error are you seeing? As long as the configuration has a type of

StringSource

, you should be able to set it as:

...
config_value:
    env: CONFIG_SOURCE
...

If there’s something that can only be set as a

String

currently, we can probably fix that to use a

StringSource

instead.

I got

Error 1: Received unexpected config entry "env_vars" at the root. Expected: "{ }".

But now that I’m thinking about it, I might be able to achieve the same thing by providing the env vars through the task definition. I’ll give that a go and see if it works

Correct - you’ll need to define your environment variables elsewhere. On the task definition, inside your image, etc.

dagster.yaml

is specifically for configuring your Dagster Instance. It’s for configuring the components of the instance (your Run Storage, Event Log Storage, etc.). There’s not a way to use it to pass arbitrary environment variables through.

separate issue: my organization doesn’t give me permission to create IAM roles which means I can’t run this through a docker-compose context. Is there anyway I can get a cloudformation script that the shows how this is built out? That could really help me work through these issues a bit quicker

https://gist.github.com/jmsanders/ee5601bd9a7648e2571cc00cd7e4b883 - this isn’t a file we plan on maintaining, but here’s the CloudFormation template that docker-compose command scaffolds for you (scrubbed of the specific vpc ids, subnet ids, and eks images it creates/pushes). I suspect you’ll need to work with the appropriate team at your organization to get certain IAM roles created anyway, but hopefully this unsticks you while you coordinate with them.

Perfect!! You’re exactly right, I don’t expect you to maintain this at all, just the example is useful because I will have to deploy via terraform/pulumi