Anyone implemented a queue (some sort of priority ...
# ask-community
g
Anyone implemented a queue (some sort of priority setup) to work with dagster? Or would you just kill and rerun to get the order right?
🤖 1
s
We use the
dagster/priority
tags to make sure that scheduled runs always take precedent over backfills, for example. Is that what you're thinking about?
g
I'm picturing ML workflows. Set off a bunch of training runs that max out resource, only to have a much better idea later. You still want to run your first batch but prioritise the new idea
s
Yeah, I think you could use priority tags to make sure things get scheduled in a certain order. (e.g. 1000 jobs from Batch 1 get queued; after 200 complete, you kick off 1000 jobs from batch 2 with a higher priority tag. Batch 2 jobs will get scheduled before batch 1.) Once dagster puts things on the queue, though, I think it's up to your execution engine to manage resources. If you had a couple of node groups in kubernetes, though, you could tag batch 2 to a different set of resources to make sure they dont overlap
❤️ 1
what ive seen with some ML-specific orchestrators is that when you kick off a job, the service actually provisions a new node group for the job, then deprovisions it afterwards. dagster cant do that AFAIK
❤️ 1
g
@Stephen Bailey really good to know on the provisioning front. Not where we are now but still something it'd be good to plan for, could equivalent functionality not be achieved with a custom DockerRunLauncher / CustomRunLauncher and GPU EC2 / cloud compute?
s
yeah, i think it probably could, you'd just have to set up those launchers beforehand, and use job config to tell your execution environment where to run them. although, im also starting to get to the edge of my knowledge base so there are probably lots of other ways to do it 🙂
❤️ 1
g
@sandy could you point me towards the best demo for using dagster to manage ML experimentation? e.g. if X1 works by some measure of success (threshold AUC or similar) spin out jobs to try X2, X3, X4 (small iterations on roughly the same experiment, or a hyperparameter tuning run if the core technique proves to be beneficial)
This is the best I've seen but API wise it's quite out of date

Thinking Machines: Running Dagster for Machine Learning Pipelines in Production - YouTube

s
Hey George - alas, we don't have a demo for the pattern that you're mentioning, although it does seem like a reasonable thing to build using Dagster
1