Hi team, has anyone seen any issues with dagster-shell and simple bash scripts terminating unexpectedly? I'm running a test with a simple script (sleep 5) and I'm seeing the attached error. One thing to note is that this works fine inside our jobs that run with our docker run launcher. The failures are only present in jobs that execute through our dagster-celery-docker executor. One other note is that ls seems to work, sleep 5 seems to fail and aws cli commands fail. I am able to get into the containers while these steps are running and execute the commands from bash without issue. I'm not seeing any docker OOM events, so I'm thinking the 137 is coming from the finally clause in the utils class in the dagster shell that is cleaning up the subprocess.

Do you have logs from one of the Docker containers that failed?

I’ll have to patch the run launcher to disable auto remove and give it a shot, the containers have been deleted by the time I can get in. I’ll give that a shot and PM them to you when I can, thank you.

Hmm it does look like 137 can be a linux exit code for OOM. Just to confirm, Docker shows the container with 137 exit code and no error in the logs?

Yes, a docker inspect of the container shows it was not oom killed. I've also read that if a sigkill is sent you can get a 128 + 9 exit code at the shell. I know if the celery worker receives a sigkill it will shutdown without waiting gracefully.

It’s strange that this issue pops up with the

celery_docker_executor

but not the

DockerRunLauncher

, since in both cases the

sleep

cli command is just running in the same docker container

I agree, it is strange. Just to confirm I re-ran all sorts of shell commands (ls, aws s3 ls, aws s3 cp, aws s3 rm, cat, sleep) all seem to work fine within docker run launcher and multi process executor. Is there something I can grep for in the worker logs that will surface what youre looking for?

So the latest is that when using

celery_docker_executor

,

ls

works but

aws

and

sleep

don’t?

We are using the docker run launcher and multiprocess executor for most jobs, we introduced the celery worker to control how certain jobs execute which is a requirement. In this case if they combine a shell script with one of our jobs that synchronizes data we need it to run in celery to control how many data synchronizers can run in parallel. We are generating our jobs from our application (they aren't hand written by developers). I think if we modify the logic in our app to generate the run config that I got working manually for jobs that require the celery executor it should work. The current issue is this now (again only happening in the celery executor):

@Dagster Bot issue shell_op is raising “DagsterInvalidMetadata on type bytes”

Created issue at: https://github.com/dagster-io/dagster/issues/8029

Thank you