Hey, I'm having some trouble with my dagster k8s instance and preemptible nodes. The dagster rabbitmq goes down every so often when we lose a node and the node affinity for the pv doesn't match. Currently we have an issue where the pv expects a label of

us-east-1a

and all the nodes are labeled

us-east-1b

or

us-east-1c

. Do I need to unpack the included rabbitmq and manually modify the statefulset or is there an easy way to facilitate this change through the top level helm chart?

Hi Kirk, this is a surprise to me- us-east-1a is the default in the rabbit-mq helm chart?

I don't see anything specifically set there, so I'm thinking zone is assumed based off of the node it gets assigned to. I don't really see any way to configure it at all. The statefulset deployment has nothing really related to it past the PVC which just sets the access mode of

ReadWriteOnce

. I'm not quite sure how that passes config to the PV

Hmm- unfortunately I haven’t seen this before. Happy to look in to their helm chart a bit but at present all I have are workarounds • Internally we use a cloud-hosted redis outside of our kubernetes cluster • We have a

k8s_job_executor

that just landed that offers pod-level isolation without celery (and no rabbitmq). Currently in experimental, would love to get feedback on it

Oh really? I'd be very interested in the k8s_job_executor. Can I use that in conjunction with celery_k8s or do I need to specify that? My use case there is I have some current k8s cron jobs that I'd like to have dagster orchestrate

What would using it in conjuction with celery_k8s mean?

I might be getting it confused - my frame of reference was Airflow with it's K8s Job Operator, which allowed you to execute arbitrary k8s jobs. I'm looking for something similar in dagster. This is completely tangential.

Gotcha- yeah our terminology can be a little confusing when you go back and forth

Related to the k8s_job_executor - do I still set envSecrets and everything else the same for it? I assume it still launches a version of the same user code repository or a set image? If there's documentation that works too

Yep! Historically we’ve offered two k8s deployments: • dagster-k8s, where we run an entire pipeline inside a single k8s job • dagster-celery-k8s, where we use celery to run each solid of the pipelne within its own k8s job

Will that include pooling or some sort of access limit?

At present we only support run-level concurrency limits, one feature gap compared to celery which enables limits per solid

PV topology stuff is not rleated to Dagster or Rabbit per se

Re:

Dagster has a native scheduler that can kick off pipeline runs on a cron schedule, and you can use whichever deployment (dagster-k8s, dagster-celery-k8s) to run the pipeline

I'm doing a poor job of explaining it, but basically I was hoping for some type of way to have dagster run an arbitrary k8s job. But the more I think about it, the more obvious it seems that I need some sort of k8s solid that would just contain a script to do something like

kubectl create job ...

Sorry, wasn't watching this thread, so okay

All good and yes it's EKS. I believe you're right on that. We use spot instances, and I think it's just whatever is available in the us-east region

and enable

balance-similar-node-groups=true

So even creating our own EBS volumes and associating them with the cluster wouldn't work right? Or is that fine but we might deal with some lag since it would be in a different AZ?

You literally can never use an EBS volume across AZs, the attach API only allows matching volume AZ and instance AZ

It seems like if we're just using spot instances for our nodes that setting up ASGs would be overkill, as we'd basically just want dedicated nodes right? I'll look into EFS

No? Not sure what you mean

Well I guess I mean since we just bid on whatever instance is available in a given region - it doesn't make sense for us to try and scale/add compute to a given AZ

Sure, if you want the broadest possible bidding pool then design wise EBS volumes are very limiting

AFAIK that's what we're working with. This is all being done on our dev/staging cluster so I think it's just the cheapest we can get it

On that I defer to you 🙂

Thanks for your help with this. This was massively helpful. I also think EFS will work for this case

The usual rule of NFS is "it will work but you'll still regret it eventually" 😉