Hey! First time user of Dagster here (been playing around with it for a couple of weeks). Something weird is currently going on with my dagster setup. Currently I'm running the dagster cloud local agent running on a VPS on GCP. Everything runs fine when I've set it up, but after a while Dagster stops working with the Python IOerror 24: Too many files open. After a bit of debugging I find that the "dagster-cloud agent run" opens a ton of sockets when running without closing them (ls -alht /proc/{$dagster_PID}/fd/ . Eventually it opens up 1024 sockets and the VPS starts erroring out. Bit like a memory leak, just with open sockets.. I've set up a @run_failure_sensor to monitor for failed jobs. This is what seems to trigger the creation of new sockets. If I disable it new sockets stop being created. Here's the "ls -alht /proc/{PID}/fd" of the dagster-agent code:

ls -alht /proc/8739/fd
total 0
lrwx------ 1 root root 64 Sep  8 13:12 13 -> 'socket:[23909]'
lrwx------ 1 root root 64 Sep  8 13:12 14 -> 'socket:[24052]'
lrwx------ 1 root root 64 Sep  8 13:12 15 -> 'socket:[23399]'
lrwx------ 1 root root 64 Sep  8 13:09 12 -> 'socket:[23156]'
lrwx------ 1 root root 64 Sep  8 13:08 11 -> 'socket:[23576]'
lrwx------ 1 root root 64 Sep  8 13:06 10 -> 'socket:[22520]'
lrwx------ 1 root root 64 Sep  8 13:03 9 -> 'socket:[21384]'
lrwx------ 1 root root 64 Sep  8 13:01 7 -> 'socket:[21333]'
lrwx------ 1 root root 64 Sep  8 13:01 8 -> 'socket:[23994]'
lrwx------ 1 root root 64 Sep  8 13:00 6 -> 'socket:[21644]'
dr-x------ 2 root root  0 Sep  8 13:00 .
lrwx------ 1 root root 64 Sep  8 13:00 0 -> /dev/pts/0
lrwx------ 1 root root 64 Sep  8 13:00 1 -> /dev/pts/0
lrwx------ 1 root root 64 Sep  8 13:00 2 -> /dev/pts/0
lrwx------ 1 root root 64 Sep  8 13:00 3 -> 'socket:[23896]'
lrwx------ 1 root root 64 Sep  8 13:00 4 -> 'anon_inode:[eventpoll]'
lrwx------ 1 root root 64 Sep  8 13:00 5 -> 'anon_inode:[eventfd]'
dr-xr-xr-x 9 root root  0 Sep  8 12:59 ..

The longer I let it run, the more sockets get created. I don't know what the next step of debugging is. I've already rebuilt the VPS once, and the same error occurred again.

Hey Alexander - we’ll take a look at this and see if we can reproduce. Whats happening within the body of the run failure sensor? Do the sockets only get opened whenever a run fails, or all the time even if no runs are happening?

@op def teams_op(context): webhook_url = "a url" msg = pymsteams.connectorcard(webhook_url) msg.title("Dagster Alert") msg.text("Task failed in Dagster.") msg.send() @job() def send_teams_msg_on_job_fail(): teams_op() @run_failure_sensor( request_job=send_teams_msg_on_job_fail, default_status=DefaultSensorStatus.RUNNING, ) def job_fail_sensor(context): run_config = {"ops": {"teams_op": {"config": {"job_name": "make_teams_alert_on_failure"}}}} return RunRequest(run_key=None, run_config=run_config)

Just to add more color here: we run dagster on AWS ECS containers and we are hitting:

E0907 21:26:00.771067497      20 <http://tcp_server_posix.cc:216]|tcp_server_posix.cc:216]>    Failed accept4: Too many open files

on the

workflows_user_code

service. We also have a bunch of sensors, and also for tasks failures. The container itself seems to get stuck after 2h of a deployment, for some random reason.

Alexander do each of those times correspond to a run? Curious if the problem is that each launched run by the sensor is leaving a socket open, even after the run finishes - I'll see if we can reproduce

No, it's not a run it's a check (which is then skipped). It's from the polling, not the run itself

got it, thanks. And those "socket:[22520]" lines - is the number in there a PID? (Might be displaying some linux ignorance here)

Don't know enough, socket FD id or something. The pid is in the ls call /proc/(pid)/fd

The other thing i'd like to check is if pressing "Redeploy" in Dagit on the Workspace tab brings the socket usage back down (that will stop and restart a subprocess on the agent)

Will test in 5

OK, that's helpful to know. (It shouldn't restart the agent process, but it will restart one of its subprocesses, which seems to be the one leaving those sockets open_)

I'm running 1.48.1 atm yeah. I'll try to downgrade and see what happens

Ah right sorry 1.48.1 not 0.48.1 - that might help explain why it started recently but downgrading dagster didn't fix it

I see that both of my servers are running 1.48.1 though. Any way of getting more verbose logging of these sensor checks or something?

I don't totally follow - 1.48.1 is the version that (may) be causing the problem

Ayy nice, that solved it actually

Ah amazing

Much appreciated Daniel 🙂

I’ve been following and will try this later today! We have rn poetry locking

grpcio = ">=1.43.0"

, so we could be grabbing

1.47.1

- this started happening last week or so.

1.48.1 is the bad one - that was released on 9/1/2022

Alright - we are also not using Dagster 1.x.x yet, so that upgrade could be useful for us too.