https://dagster.io/ logo
#dagster-plus
Title
# dagster-plus
j

John Smith

07/25/2023, 12:48 PM
Hello 1. Can my dagster cloud be configured with a "warm pool" feature which keeps a pre-specified number of Fargate processes running to accept short running tasks that need quick turnaround? < 10 seconds startup time 2. if not, I'm considering running dagster hybrid using ECS + EC2 warm pools to achieve this, but would like to better understand what potential challenges I'd face using EC2 instead of Fargate - presumably elementl already performed this analysis and decided on fargate. could you share your reasons? 3. Can dagster "recycle" processes that share the same code envs to avoid setup and teardown costs? I guess the
in_process_executor
somewhat achieves that for an entire job but hoping for other 'clever' options. 4. is support for hybrid agents for AWS lambda on roadmap?
m

Matt Clarke

07/25/2023, 1:09 PM
This is none of the above, but I've been experimenting with SOCI indexes to speed up cold starts. In short, it allows lazy loading of images, which depending on the size of your containers might give you a decent speed up. I doubt you'd get sub 10 seconds, but there's probably some extra performance in there, and little to no downside https://aws.amazon.com/blogs/containers/under-the-hood-lazy-loading-container-images-with-seekable-oci-and-aws-fargate/
❤️ 1
j

jordan

07/25/2023, 1:17 PM
Short answer is that 1 isn’t something we’ve built support for out of the box. We talked about it, but ultimately went with https://dagster.io/blog/fast-deploys-with-pex-and-docker instead (some of the tradeoffs are discussed in that blog post) 2 should work - we have plenty of users who use EC2 instead of Fargate. We use Fargate by default because of the lower operational burden. 3 would require you to write your own executor. And 4 isn’t currently on the roadmap. A potential challenge given each lambda execution’s limited lifespan would be keeping the code servers around - unless it’s something like “lambda and ec2” or “lambda and ecs”, we potentially trade off the run startup time for the code server startup time. One thing that is on the roadmap (but the specifics of what it will mean are still very hazy) is better support for lower latency/steaming patterns. We’ve found that the more we squint at the low latency challenge, the more it starts to look like streaming/micro-batching. Plus I was about to post what Matt posted but he beat me to it.
❤️ 1
d

Daniel Gafni

07/25/2023, 1:25 PM
It seems like you need the Celery executor? This way the Celery worker would always be running and ready to accept tasks to execute
j

John Smith

07/25/2023, 2:01 PM
Would the celery executor "play nicely" with a dagster hybrid ECS deployment?
Thanks @jordan can I infer from that post that assuming my env dependencies are entirely python and can entirley be managed with pip and virtualenvs, I should focus on bundling my code locations as pex / shared paths rather than creating a docker image for each of them? And as a wider corollary, I don't even need to concern myself with docker if running on dagster cloud / hybrid.
j

jordan

07/25/2023, 3:55 PM
celery isn’t supported with cloud; the agent won’t manage the celery worker lifecycle (whereas it will manage code server lifecycle) we’d need to prioritize some work to make PEX available on hybrid agents (instead of just serverless) - but that work is probably a smaller lift and more likely to happen than warm pools/lambda executors. can i put you in touch with @Dagster Jarred to hear a bit more about your use case? i think it might help inform some of our decisions around lower latency execution.
d

Dagster Jarred

07/25/2023, 4:32 PM
@jordan would the fully isolated sensors project potentially help address this? i guess you still have container start up time even if you can spin up jobs substantially faster in that world.
j

John Smith

07/25/2023, 4:40 PM
Just to clarify the question about pex and not wanting to deal with docker is orthogonal to the low latency case. Many of my end users are novice users who are familiar with pip but less so with docker.
m

Matt Clarke

07/25/2023, 5:02 PM
@jordan is there any movement for pex for hybrid deployments? We’d love to make use of it, but afaik it’s only supported for serverless deployments?
j

jordan

07/25/2023, 7:46 PM
It’s not something we were planning to do any time soon, but in part because we hadn’t really heard much of a request for it. Given the interest in this thread, I’ll see if it’s something we can squeeze onto my team’s Q3 roadmap.
1
j

John Smith

07/27/2023, 5:12 PM
@jordan @Dagster Jarred can I clarify what "fully isolated sensors project" was intending to address? sped up startup times?
2 Views