set the channel description: Get the latest updates on using Dagster for ML, ask questions, and discuss with other members of the community.

set the channel topic: Using Dagster for machine learning

Hi everyone! Welcome to the machine learning channel!

Good timing 🙂 We are starting to use Dagster for our ML pipelines, and one of the questions that came up was whether we should implement them as Jobs or Assets. In many cases, we run various experiments and control the run parameters manually. I can easily see how to use Jobs for that, but I can’t see how to use assets since these are not pre-defined partitions or anything like that. Any thoughts on this?

This was a good idea

👋 Does anyone have recommendations on a platform for online pipeline deployment with integration with ML models? What I mean by a platform for online pipeline deployment with ML: • Real-time, not batch workflows • Can do pure data processing using API (i.e., input, return output) • Can do ML prediction + Downstream Data Processing using API (i.e., input, prediction with ML model, perform data processing, return output) • Data/Model versioning is nice to have • Experiment Tracking is nice to have

Hey everyone! We are building a forecasting system at my company and plan to build on top of Dagster. The 2 main areas where I have to build up my knowledge are: 1. Partitioned jobs w/ dbt (for backtesting forecasting models) 2. Heterogeneous execution resources (specifying compute resources per op / asset) I'm delighted ye are giving ML it's own channel @sandy as I've been working with Dagster in ML context for a few years!

Hi everyone, we just released a new guide on managing ML models. https://docs.dagster.io/guides/dagster/managing-ml

sharing this thread here - in case anyone has had experience doing idempotent batch inferences using dagster... How to avoid circular dependencies?

Hello - keen to know where we are in terms of supporting offloading asset materialization to cloud hosted runners (I'm thinking GCP App Run / Azure Container Apps / etc..) My us-case may not be the most common one, but I'm sure you'll see parallels to other scenarios: I have an asset

data

that gets updated daily. I have a statically partitioned asset

model

that contains ~100 different models, all of which can make predictions on that data The task: I have another statically partitioned asset

shap_values

that take in data and one model and generate the matching shap values (slow-ish computation) - so this requires ~100 asset materializations This task is embarrassingly parallel, it should be possible to run all 100 models at once to generate the various

shap_values

However, this requires a good amount of compute, so an ephemeral runner would be ideal. Thoughts?

Is there a compare and contrast out there for Dagster and Aqueduct?

Has anybody used Dagster to orchestrate distributed training in k8s successfully? Basically this requires to start multiple pods per one op. Any advice except using the k8s_job_op? I’m thinking about hacking the dynamic graph to spawn multiple pods with .map

Hey Folks - I hope this question resonates a bit better with folks here. We are starting to to use dagster for both our data engineering (ingestion / scraping jobs) and downstream ML models that depend on those data. I have an IO manager written that can allow me to select a subset of ops in a graph and provide config on the root op to determine (for the IO manager) where to load an input from (as it is now disconnected). This was based heavily on the ideas outlined here. This is really helpful when experimenting on ML pipelines because I can allow users to configure how they get data from the UI in dagit as opposed to changing source code between runs. This means we can iterate on different parts of a pipeline in parallel (nobody is waiting for upstream parts to be built out if they can provide a csv or a flat file of some sort of transformed data as we support it in our IO manager). Additionally, this can make debugging production jobs much faster Has anybody implemented anything like this but for assets specifically? My key requirement here is to be able to select a single asset in dagit and materialize it while providing config for the run that will allow my IO manager load input from somewhere other than where the upstream asset is stored and effectively stub the upstream assets for my selected asset. I've asked about this here before but I don't understand how to achieve what I want with assets specifically

Does anyone know of or has done a recent evaluation of Dagster against Kubeflow? I am working on the transition and would like to learn from experiences and thoughts.

We just published a blog post on fine-tuning LLMs with Dagster. https://dagster.io/blog/finetuning-llms