hey folks! I’m working on a project that will do most of the heavy processing outside of dagster (think API requests to a cloud function, SQL queries to snowflake, etc) How hard can we push dagster with that setup? For example, can I have multiple jobs running in 1 cpu core at the same time? Would I/O bound operations like API requests block a process? What’s the best pattern to follow in this case?

I think the process initialization costs and fixed memory overheads will probably be the limiting factor in trying to run many processes at the same time with limited CPU (and I assume memory).

gotcha! in this case I assume the number of concurrent dagster jobs is much bigger than the number of physical cores in the cpu? like 30+ dagster jobs in an 8 core cpu vm

Yep, you may end up with some slow downs when the processes contend for CPU resources but if the majority of the time is spent waiting on IO then that shouldn’t be often.

sounds good