Fun weekend with dagster and mlflow I try to integrate mlflow with dagster pipelines. I will publish an architecture of it. I am mixing: • mlflow + minio (S3) + postgres for registering the model • dagster with daemon launching specific mlflow image with a scheduler and amazing sensors (simple but the idea is here). If you have any improvements tell me 🙂 https://github.com/slamer59/dagster-mlflow.git

I don't think you would need MLFlow given the AssetMaterialization feature dagster provides. Here is why: • A dagster pipeline is the same in principle as an experiment in MLFlow. • Dagster runs are the same as ML Flow runs. • Dagster assets are the same as MLFlow artifacts. In principle, to reproduce a model run, all you have to do is rerun a pipeline in Dagster with the appropriate asset materializations.

Thanks tour feedback. • Pipelines in dagster is more than mlflow experiments. Dagster gives a deeper reusability IMHO • Mlflow as a good abstraction for many frameworks, deployment/serve, etc. Similarities but also differences. Dagster is more focus on data lifecycle where mlflow on the models life cycle. So agree that some features are similar.

i'm not trying to suggest that they are somehow completely similar. It's just that all the features related to artifact versioning (which includes models, plots, data) that MLFlow currently provides are all things Dagster can do through asset materializations. MLFlow model artifact versioning can be completely replaced by Dagster AssetMaterialization with no loss in functionality.