Conversely, have folks attempted to attach live debuggers to their dagster jobs running on k8s?

I cant think of anything dagster specific that would change the normal level of shenanigans required to attach in k8s. you likely need to have the

SYS_PTRACE

securityContext

capability

set https://github.com/benfred/py-spy#how-do-i-run-py-spy-in-kubernetes whether you are running the default multiprocess executor or the k8s job executor will effect what process you are hunting for if its the underlying op code you are hoping to get at

thanks for the response @alex . My hope was that there would be a way to inject a command into a dagster container. For example if I were able to do

python -m debugpy --listen 5678 dagster ...

I should be able to attach a remote debugger.

hmm, not sure of a good way to alter the entry point currently. You could opt to add the

debugpy.listen

call in python, maybe gated by an env var. I personally recently successfully used

madbg

to attach in k8s https://github.com/kmaork/madbg