Cameron Gallivan

09/15/2021, 12:22 AM
Any runs that we’re in progress freeze and will be stuck in the starting/started phase even after a server restart and need to manually be terminated. The deployed containers ignore
docker-compose down
and the command times out as well. This seems to only happen during sustained higher load times but I’m not sure if that is the sole/exclusive cause?


09/15/2021, 2:45 PM
Hi Cameron - could you share more about where your postgres DB is running / any resource limits it has? It does seem like its getting overloaded, curious if that's a problem with Dagster or indicative that you need to allocate more resources for your database (possibly both)

Cameron Gallivan

09/15/2021, 5:36 PM
It’s on the same EC2 instance as a docker container, this is built on top of the docker-compose example from the docs (at least what it was in ~v0.9). I don’t think I’ve made made any modifications to the postgres container other than having it mount the data to a docker volume but this problem happened prior to that. Some more details: The EC2 instance itself is a relatively small one, a m4.xlarge with 4 vCPUs, and i’d like to keep that the same if possible. If it is just a resource bottleneck is it feasible to lower the pipelines max concurrency (currently capped at 25) and somehow increase the resources allocated to the postgres DB container?