👋 Hello all, catching up a bit on run isolation. It looks like a really cool feature, but I have a few questions: 1. Will there be reduced cost in the future for using non-isolated runs outside of the faster startup time? Seems like going from 4cpu, 16gb ram -> .25cpu, 1gb ram should be cheaper, but, perhaps I’m misunderstanding how this works together 🤔 a. EG: Non-isolated runs should be much cheaper for short running jobs, where lots of the cost will be from the ~3 minute startup time. However, many of our jobs are in the 30+ minutes to 2 hours range. Currently, we don’t have much a reason to want to switch over since the dagster cost is a small portion of the overall job cost, and we have still other jobs are running and taking advantage of the increased compute resources of isolated runs (even if this isn’t ‘best practice’). 2. Do we have access to said PEX files? Something I’m looking into is creating a PexStepLauncher and a great starting point could be to just deploy said pex objects to an EC2/ECS instance. This is almost like a “psuedo” hybrid deployment :blob_thinking: Thanks!

Hi Zach! For 1 - one thing that is worth mentioning is that you don't pay for run startup time, in either non-isolated or isolated runs. You only pay for the step execution time, i.e. the time it takes to run your actual code - so those 3 minutes you mentioned aren't a factor in pricing. We weren't thinking of cost reduction as the main reason to use non-isolated runs, but rather faster iteration speed and lower latency, especially while you're iterating on changes to your jobs. That doesn't negate your point about pricing but might give some context on why you might choose to use a non-isolated run. 2 - No direct access to the PEX files that serverless uses currently, although we are thinking of bringing PEX builds to the hybrid agents as well, at which point I think all the code you would need to build and run Dagster code using a pex file would be open source.

Thanks Daniel that makes sense & that ks for the clarification! Figured there may be some more behind the scenes changes to actually make non-isolated runs actually cheaper to host anyways :) For context, were scoping out rolling dagster to other teams that may not always use spark, so using something like Non-isolated PEX -> ecs step launcher sounds like a good use case where as right now we have lots of people doing work inside of the dagster instances themselves. We alot of testing locally so the cicd wait time isn't killing us ATM I definitely like the idea of being more efficient with dagster resources and kind of having it push us towards more loosely coupled set ups however :) P.S: Noticed your slack status -- hope you get feeling better 🙂