The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

image.png

A few questions about the Serverless spin up times:

1. Is there an SLA or otherwise a known maximum time tasks will take to spin up? Based on my experience so far, the lag is around about 2 mins on average. 
2. Are there any plans to reduce the task spin up time at all? It seems rather slow and it would be great if this was _at least_ under a minute


Hi Todd - we currently use AWS Fargate and unfortunately these can be typical Fargate spin up times for new containers. Having low latency startup options is on our roadmap. If your runs are lightweight, you can look at using <https://docs.dagster.io/dagster-cloud/deployment/serverless#run-isolation|non-isolated runs> which execute the run on the code server, rather than spinning up a new container.

Thanks <@U03FY5SQWKH>. That does potentially look useful. However it only appears to be available at the deployment level? Not at an individual job or op level? 

We only have one op/job where this is a potential concern. So the goal would be to an use non-isolated env for this on op and then isolated for the rest. 

But it feels like this isn’t presently possible? 

You can attach the run tag `"dagster/isolation": "disabled"` to your job and it should run non-isolated. This was recently enabled and is not documented yet. Let us know if you see issues doing this.

Ah nice! I’ll give that a whirl. Thanks for the swift reply and advice, Shalabh :) 

Hi <@U03FY5SQWKH>,

I’ve set the job tag as you’ve suggested, and as you can see in the first log, the op is being executed without an isolated environment.

However, the op is still waiting for compute resources to spin up. Which is a bit puzzling and either the first log is wrong, or the op the incorrectly spinning up / waiting for resources.

Any ideas?

```@job(
    tags={
        "dagster/priority": DagsterQueuePriority.HIGH_PRIORITY.value,
        "dagster/isolation": "disabled",
    },
)
def alerts():
    alert()```

Oh we'll have to look into why this isn't working as expected. Can you try adding the tag using the schedule or sensor that launches this job? Eg for a sensor see <https://dagster.slack.com/archives/C02LJ7G0LAZ/p1691084764927709?thread_ts=1691071465.064239&amp;cid=C02LJ7G0LAZ>

<@U03FY5SQWKH> No joy sorry
```    ScheduleDefinition(
        job=alerts.alerts,
        cron_schedule="*/5 * * * *",
        tags={"dagster/isolation": "disabled"},
    ),```


I'll take a look. Are you able to run this job non-isolated using the UI checkbox?

Ah that indicates something else is going on with this run. I can take a look at our end - can you please share your org name (dm is ok too).