A few questions about the Serverless spin up times...
# dagster-serverless
A few questions about the Serverless spin up times: 1. Is there an SLA or otherwise a known maximum time tasks will take to spin up? Based on my experience so far, the lag is around about 2 mins on average. 2. Are there any plans to reduce the task spin up time at all? It seems rather slow and it would be great if this was at least under a minute
Hi Todd - we currently use AWS Fargate and unfortunately these can be typical Fargate spin up times for new containers. Having low latency startup options is on our roadmap. If your runs are lightweight, you can look at using non-isolated runs which execute the run on the code server, rather than spinning up a new container.
Thanks @Shalabh Chaturvedi. That does potentially look useful. However it only appears to be available at the deployment level? Not at an individual job or op level? We only have one op/job where this is a potential concern. So the goal would be to an use non-isolated env for this on op and then isolated for the rest. But it feels like this isn’t presently possible?
You can attach the run tag
"dagster/isolation": "disabled"
to your job and it should run non-isolated. This was recently enabled and is not documented yet. Let us know if you see issues doing this.
Ah nice! I’ll give that a whirl. Thanks for the swift reply and advice, Shalabh :)
Hi @Shalabh Chaturvedi, I’ve set the job tag as you’ve suggested, and as you can see in the first log, the op is being executed without an isolated environment. However, the op is still waiting for compute resources to spin up. Which is a bit puzzling and either the first log is wrong, or the op the incorrectly spinning up / waiting for resources. Any ideas?
Copy code
        "dagster/priority": DagsterQueuePriority.HIGH_PRIORITY.value,
        "dagster/isolation": "disabled",
def alerts():
We are on 1.4.5 BTW
Oh we'll have to look into why this isn't working as expected. Can you try adding the tag using the schedule or sensor that launches this job? Eg for a sensor see https://dagster.slack.com/archives/C02LJ7G0LAZ/p1691084764927709?thread_ts=1691071465.064239&cid=C02LJ7G0LAZ
@Shalabh Chaturvedi No joy sorry
Copy code
        cron_schedule="*/5 * * * *",
        tags={"dagster/isolation": "disabled"},
I'll take a look. Are you able to run this job non-isolated using the UI checkbox?
@Shalabh Chaturvedi, nope. Same result.
Ah that indicates something else is going on with this run. I can take a look at our end - can you please share your org name (dm is ok too).