🧚 dancing 🕺 dancing 💃 dancing 🧚‍♀️ Hello guys, My project's current status I am working on the development of an ML product. So far we have used DVC to define our ML pipeline, including preprocessing steps, training, baselines computations, and so on... We are working with GCP and so far we are running our pipeline using vertexAI training jobs. Where I need your help Now we want to move to another DAG framework that allows us to run the different steps on different machines (with or without GPU) and that is more integrated with

GCP

. I have always considered

kubeflow

as a strong option as it it well integrated with

GCP

and specifically

VertexAI

. Now I saw a talk at Pycon Italy recently about

Dagster

and I felt like it could compete with

Kubeflow

but I still can't figure out whether

Dagster

is better. One specific question I don't have the answer for is if there is an easy way to run the

Dagster

ops on

Cloud Run

or

VertexAI

. Thanks a lot for your help 🙂 your project looks really cool!!! (@Andrea Giardini I saw you at pycon.italy, and contacted you on linkedin, thought that would make sense to ask my question here 🙂 ) 🧚 dancing 🕺 dancing 💃 dancing 🧚‍♀️

I used to use Argo Workflows for our orchestrator but chose to use Dagster for my current company. Can't specifically speak to Kubeflow, but here are some of the tradeoffs in my mind: Argo Workflows, and I'm assuming Kubeflow, is very nice in that it's very "cloud-native". In Argo Workflows at least, the framework was pretty light and felt just like other Kubernetes resources. You could write whatever code you wanted -- all it cares about is input/output. A downside is that it was very hard for me to train up other data team members on it, because you have to understand not just the framework, but also Kubernetes. Dagster has a much cleaner separation of the execution and code environment. It's built from the bottom up to support the writing of arbitrary Python code without understanding the execution environment, although the backends (kubernetes, ECS, Docker) are quite flexible. So if you are asking other people to write code and pipelines that have non-trivial logic, the Dagster framework provides a lot of functionality to make things coherent. However, you also have to implement a lot of "dagster-specific" functionality, classes, etc. WRT to Vertex AI, I'm not sure how the integration works, but if they are running the training/etc., the pattern I have seen is that you can have a Dagster job that kicks off training pipelines with a variety of config settings, then polls for responses as to the completion. In that case, though, it's sort of a glorified cron job, and I'm not sure how valuable that woudl be for your use case

Thanks a lot for the answer Stephen! I was also wondering if there is an integration with google cloud in Dagster, like if there is a way to specify some Machine specifications to run an op on, like through a decorator or something like that?

Hello @Adrien Ruault 👋 Happy to hear you enjoyed our talk at Pycon 🙂 I didn't have the chance to use VertexAI yet unfortunately. Regarding Dagster, it's definitely possible to schedule every op / graph on different machines, based on the resources requested. Generally, you can use the

dagster-k8s/config

tag to add all kinds of Kubernetes specifications (think volumes, requests/limits, env vars, etc.). You would specify all that in the decorator. Over time, we started building our own abstractions on top of the classic

op

to make certain operations easier, but you don't need do to that from the beginning. It's more of a way to abstract Kubernetes away from our developers and data scientists. Happy to have a chat in case you want to know more about it 🙂

Hi - @Andrea Giardini. I was browsing through this channel and this conversation caught my attention. I currently work at a robotics startup where we use Dagster for orchestrating our batch jobs for building up our deep learning datasets. I've ended up developing a simple framework and abstraction layer to make it easy to run arbitrary Kubernetes jobs in a pipeline. I'd be curious to hear what your team has built to simplify the developer experience!

@Arslan Aziz Sure thing! I wrote you a DM

Hi @Andrea Giardini , After reading this thread I am more interested in your talk at Pycon Italy. We are outlining the approach that we would like for our ML pipelines in the team and the tools to use for that. I, personally, have my eyes set on Dagster as my choice. Would love to hear your experience using Dagster for MLOps, how far are you, what are the pros/cons that you have discovered etc.

Here you go @Akshay Verma

Hi @Andrea Giardini Thanks. One question that arises for me is, Did you look into Vertex AI as well for pipelines? What I can assume is that people would say it is better integrated in the GCP ecosystem.

No, we did not look at Vertex AI