Hey everyone! I've got a question around versioning in Dagster. Say we have a job that goes op A -> op B -> op C. And op B runs a model. Is there a way to say that op B should run a specific image or version of our model code? The reason we might get into this case is that as we do model development (what is contained in op B) we will want to test changes on instances with enough compute to get meaningful results (i.e. not locally). We might have several new models we would want to test at once before moving those changes into main. There are certain concurrency limits we will need to enforce across all ops with specific tags so it would be great if we can keep this in one deployment. We are likely using k8s as our orchestration for dagster, potentially in tandem with celery to enforce some of those concurrency limits across ops+tags. Are there any patterns in dagster to enable a workflow like this? Any docs I should read?

Just so that I’m clear, are you looking for a way to construct a job that allows you to do parallel evaluations of certain model candidates? Or are you looking for a way to construct a job that allows you to abstract away what the currently selected model is (for production)?

I'm looking for a way to construct a job through a single deployment where some of the ops in that job are running different versions of code or different images. Or to run versions of jobs that aren't the version of the code dagster is running.

Ah, I see… running different images per op is not possible at the moment. Feel free to track progress (and upvote the issue) here: https://github.com/dagster-io/dagster/issues/4277

(Hi, I work with Amadou.) Would using different code locations with different ECR images / built from different git working branches be a way to manage this? If you add two different code locations with the same repo/job definitions, does that cause any errors? I'm imagining that Amadou and I could both have working branches with different versions of op B, running different models. I build mine and tag it

dagster-model-mfickett:latest

and add a code location, and run my version of

model_job

. Amadou builds his version and tags it

dagster-model-acrookes:latest

and adds a code location, and runs his version of

model_job

(same job name but from a different code location). And then eventually we merge to main and build/tag

dagster-model-prod:latest

which goes to a third code location which defines the production

model_job

which we run on a schedule. Would that work?

That would work, yes. I misunderstood the original question, which I thought was around running different code versions in the same run of a job. At the point that you have different code locations, you are essentially running different jobs in different repos. You’d have the Mark version of the repo, the Amadou version of the repo, and the production version of the repo.

And then what is the best way to run those jobs once we have the different images? Would it be to use the k8s image override mentioned at the bottom of the issue you linked above or is there another approach you would recommend?