Hello 1. Can my dagster cloud be configured with a "warm pool" feature which keeps a pre-specified number of Fargate processes running to accept short running tasks that need quick turnaround? < 10 seconds startup time 2. if not, I'm considering running dagster hybrid using ECS + EC2 warm pools to achieve this, but would like to better understand what potential challenges I'd face using EC2 instead of Fargate - presumably elementl already performed this analysis and decided on fargate. could you share your reasons? 3. Can dagster "recycle" processes that share the same code envs to avoid setup and teardown costs? I guess the

in_process_executor

somewhat achieves that for an entire job but hoping for other 'clever' options. 4. is support for hybrid agents for AWS lambda on roadmap?

This is none of the above, but I've been experimenting with SOCI indexes to speed up cold starts. In short, it allows lazy loading of images, which depending on the size of your containers might give you a decent speed up. I doubt you'd get sub 10 seconds, but there's probably some extra performance in there, and little to no downside https://aws.amazon.com/blogs/containers/under-the-hood-lazy-loading-container-images-with-seekable-oci-and-aws-fargate/

Short answer is that 1 isn’t something we’ve built support for out of the box. We talked about it, but ultimately went with https://dagster.io/blog/fast-deploys-with-pex-and-docker instead (some of the tradeoffs are discussed in that blog post) 2 should work - we have plenty of users who use EC2 instead of Fargate. We use Fargate by default because of the lower operational burden. 3 would require you to write your own executor. And 4 isn’t currently on the roadmap. A potential challenge given each lambda execution’s limited lifespan would be keeping the code servers around - unless it’s something like “lambda and ec2” or “lambda and ecs”, we potentially trade off the run startup time for the code server startup time. One thing that is on the roadmap (but the specifics of what it will mean are still very hazy) is better support for lower latency/steaming patterns. We’ve found that the more we squint at the low latency challenge, the more it starts to look like streaming/micro-batching. Plus I was about to post what Matt posted but he beat me to it.

It seems like you need the Celery executor? This way the Celery worker would always be running and ready to accept tasks to execute

Would the celery executor "play nicely" with a dagster hybrid ECS deployment?

celery isn’t supported with cloud; the agent won’t manage the celery worker lifecycle (whereas it will manage code server lifecycle) we’d need to prioritize some work to make PEX available on hybrid agents (instead of just serverless) - but that work is probably a smaller lift and more likely to happen than warm pools/lambda executors. can i put you in touch with @Dagster Jarred to hear a bit more about your use case? i think it might help inform some of our decisions around lower latency execution.

@jordan would the fully isolated sensors project potentially help address this? i guess you still have container start up time even if you can spin up jobs substantially faster in that world.

Just to clarify the question about pex and not wanting to deal with docker is orthogonal to the low latency case. Many of my end users are novice users who are familiar with pip but less so with docker.

@jordan is there any movement for pex for hybrid deployments? We’d love to make use of it, but afaik it’s only supported for serverless deployments?

It’s not something we were planning to do any time soon, but in part because we hadn’t really heard much of a request for it. Given the interest in this thread, I’ll see if it’s something we can squeeze onto my team’s Q3 roadmap.

@jordan @Dagster Jarred can I clarify what "fully isolated sensors project" was intending to address? sped up startup times?