memory availability for the job runners (update: on Cloud Serverless) seems to be fluctuating quite a bit. it’s typically ~32GB, but I got 3.8GB this on this one and it causes the job to fail:

Hey Alex - is this using Dagster Cloud serverless?

yes.

you can force it to be 32GB by checking the "Isolate Run Environment" checkbox in the Launchpad (the tradeoff is that the run will take a bit longer to start as it sets up its own isolated environment with 32GB). More information on this here: https://docs.dagster.io/dagster-cloud/deployment/serverless#run-isolation

thank you!

No problem - just checked and I believe the limit is actually 16GB not 32GB as per https://docs.dagster.io/dagster-cloud/deployment/serverless#limitations

yep, I see that now… did not see that doc before. FYI, this is the code that gives me 32GB total memory for most runs:

def report_memory_usage(logger: logging.Logger):
    import psutil
    import os
    process = psutil.Process(os.getpid())
    total_mem = psutil.virtual_memory().total / 1024 / 1024
    available_mem = psutil.virtual_memory().available / 1024 / 1024
    process_mem = process.memory_info().rss / 1024 / 1024
    report_info = f'process memory usage: {process_mem} MB / total: {total_mem} MB / available: {available_mem} MB'
    <http://logger.info|logger.info>(report_info)

I don't think there's currently a way to change the default (runs that are triggered by a schedule or sensor will always be isolated)

gotcha. thanks. been doing it manually for now.

Oh wait, yes there is:

non_isolated_runs:
  enabled: False

the default value for that is False, and my config does not specify it.

Ah yeah, the problem is the docs. Will fix, thanks!

thank you much! go Dagster!

that's interesting that psutil is saying 32GB are available - we're running in ECS and are definitely setting the memory to 16384. It's possible that it gives the process more memory but still kills it if it exceeds the set limit

I’m not sure about ECS and

psutil

specifically - but one potential misreporting problem is that you are getting the resource information of the host machine and not your containers allocation of that resource