https://dagster.io/ logo
#dagster-cloud
Title
# dagster-cloud
j

Josh Alexy

02/11/2023, 8:33 PM
is there a correlation between the image size and how fast the image boots? I had pretty fast image boot times in the serverless environment, then I added a DAG that has a pretty large dependency. Now the boot times for isolated runs are very long. If this is the case -- what can I do to prevent this? ie, isolating resources for the heavy job?
j

jordan

02/13/2023, 2:49 PM
There is indeed - serverless runs on AWS Fargate which doesn’t currently support image caching: https://github.com/aws/containers-roadmap/issues/696 That means that each time you launch a run, there’s an image pull happening. The larger the image, the longer the boot time. One mitigation we now support for this is to deploy your code as a PEX file (basically a zipped file of Python dependencies) instead of a docker image: https://docs.dagster.io/dagster-cloud/deployment/serverless#customizing-the-runtime-environment It’s the default behavior for any new serverless deployments and it generally reduces both the build and startup time pretty nicely. Alternatively, if you do want to stick to building docker images, then yes, building a separate image for the heavy job will at least help limit the impact to only that one job.