Anyone implemented a queue (some sort of priority setup) to work with dagster? Or would you just kill and rerun to get the order right?

We use the

dagster/priority

tags to make sure that scheduled runs always take precedent over backfills, for example. Is that what you're thinking about?

I'm picturing ML workflows. Set off a bunch of training runs that max out resource, only to have a much better idea later. You still want to run your first batch but prioritise the new idea

Yeah, I think you could use priority tags to make sure things get scheduled in a certain order. (e.g. 1000 jobs from Batch 1 get queued; after 200 complete, you kick off 1000 jobs from batch 2 with a higher priority tag. Batch 2 jobs will get scheduled before batch 1.) Once dagster puts things on the queue, though, I think it's up to your execution engine to manage resources. If you had a couple of node groups in kubernetes, though, you could tag batch 2 to a different set of resources to make sure they dont overlap

@Stephen Bailey really good to know on the provisioning front. Not where we are now but still something it'd be good to plan for, could equivalent functionality not be achieved with a custom DockerRunLauncher / CustomRunLauncher and GPU EC2 / cloud compute?

yeah, i think it probably could, you'd just have to set up those launchers beforehand, and use job config to tell your execution environment where to run them. although, im also starting to get to the edge of my knowledge base so there are probably lots of other ways to do it 🙂

@sandy could you point me towards the best demo for using dagster to manage ML experimentation? e.g. if X1 works by some measure of success (threshold AUC or similar) spin out jobs to try X2, X3, X4 (small iterations on roughly the same experiment, or a hyperparameter tuning run if the core technique proves to be beneficial)

Hey George - alas, we don't have a demo for the pattern that you're mentioning, although it does seem like a reasonable thing to build using Dagster