What's the relationship between a repo/job's specified executor and a Dagster Cloud agent's type? Like if I had a Kubernetes agent running and specified a multiprocessing executor on the job, that would be nonsensical.

Like if I had a Kubernetes agent running and specified a multiprocessing executor on the job, that would be nonsensical.

Having a K8s agent means that the

user code servers

managed by the agent are k8s deployments, and likely that each run is its own k8s job. Using the multiprocess executor here makes sense - the pod created for the run job would create sub processes in that container for each op.

Sort of. That specific example does. I think I'm confused about how agents actually cause steps to be run. The overall picture of the agent getting a job and then making it run somehow seems clear. But I'm having trouble following things like: with Kubernetes, will 1 job run on one pod (what you described), or use 1 ephemeral pod per step within the job (k8s_job_executor's description)?

I’m having trouble following things like: with Kubernetes, will 1 job run on one pod (what you described), or use 1 ephemeral pod per step within the job (k8s_job_executor’s description)?

This is configurable per job, by setting the

executor_def

on the job. So by default you get an executor that can toggle between

in process

and

multiproces

via config but in your case youll want to set it to

k8s_job_executor

to achieve your desired EKS/fargate step per k8s pod set-up

Gocha, thank you!

Keep in mind that if each step runs as a different pod, you’ll probably need to store the outputs in a shared location. I haven’t tried that, but you’ll need to store them in a bucket or DB. I think that by default they are store on the local disk, unless they are materialized as assets (or something like that..)

Yeah, I've tried out some cloud jobs with s3 output storage, which works pretty seamlessly.

a k8s agent and an ECS agent?

the k8s agent summons containers by interacting with k8s cluster, using k8s primitives (Deployment, Job, etc) the ECS agent summons containers by interacting with AWS, using AWS ECS primitives (Task, etc)