I have a question regarding the Docker Executor. Currently I have Dagster deployed with docker on a single beef server. I also have 2 code locations running in the same server. I’m using the Docker Executor to launch jobs. I would like to move the Dagster deployment to K8s. I would also move the code locations there. However, I still want to run the jobs on the beefy server (as docker containers). Would setting the DOCKER_HOST environment variable in the code location container be enough to achieve this behavior?

Hey Daniel - Kubernetes doesn't have great support for giving you direct access to a docker socket unfortunately in my experience - so I think ti would take a lot of work to get the DockerRunLauncher working in k8s (and may not be possible at all in some clusters)

Oh yeah… totally forgot about it. Is there anything I can do about this problem at all? Maybe with Celery? On a side note, seems like using podman would work here… it’s not like it’s possible right now, but maybe in the future a Podman Executor would be a nice addition to Dagster.

why do you want to use kubernetes?

I need high availability for Dagster and for some of the jobs. The server is not very reliable. The other jobs can (and should) still be running on the bare metal server. Some of the other infrastructure I already have is running in K8s, Dagster is the only thing left on the server.

we could probably augment the docker run launcher to be able to run a docker client running on a different host

Should be straightforward right?

Ok yeah I understand your original question about DOCKER_HOST now

Ok cool!

The former (possibly dagit too for manual runs if you’re not using the run queue)