https://dagster.io/ logo
#dagster-ecs
Title
# dagster-ecs
m

Mitchell Hynes

01/30/2023, 11:31 PM
The start time for my tasks range from 20-60 seconds. Does anyone have any Dagster-specific* advice on ways to get this number down?
m

Mike Atlas

01/30/2023, 11:49 PM
You can try to shrink your container image (we use
python:3.10-slim-bullseye
as a base) to shave some cold-boot time off, but I'm not sure this is something Dagster can fix, presuming you're using ECS with Fargate.
👍 1
I think if you use ECS with EC2, you can set up image caching and that would speed up startup
You might even get some benefit using ECR public gallery for the base image, eg:
public.ecr.aws/docker/library/python:slim-bullseye
https://gallery.ecr.aws/docker/library/python
m

Mitchell Hynes

01/30/2023, 11:57 PM
I’m not sure this is something Dagster can fix, presuming you’re using ECS with Fargate.
Ohh yeah, this is not dagster’s fault ofc. But it’s good to ask in here regardless 🙂
m

Mike Atlas

01/31/2023, 4:32 PM
Now that I think on it, using the ECR public gallery base image isn't going to help with deployment/cold boot times from Fargate, only help with build times for your image. The extra layers on the image to make it customized with your user code have to be pulled from your (private ECR) repo anyways, which is also within the aws intranet, so the network transfer speed is really about the image size and not public/private ECR base image.
4 Views