The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hello, i have a question similar (identical really :slightly_smiling_face: ) to <https://stackoverflow.com/q/61330816/1919374|this SO> from 2020. I think from what i can see that maybe the api has changed or evolved a bit since then so i'm wondering if anyone would have any pointers or if there are any examples or tutorials using more recent versions.

my use case is i want to build some anomaly detection pipelines for batches of metrics.

eg you just define some sql for each metric and then dagster handles all ingest, train, score pipelines dynamically, one for each metric.

so the sql question in the SO above is a good starter for the ingest part of my pipeline for example.

(i have built an <https://github.com/andrewm4894/airflow-provider-anomaly-detection|airflow anomaly detection provider> that does something similar but now looking to make maybe similar project that does a bit more and uses dagster to orchestrate instead of airflow)

I think the answer would be pretty much the same as the SO answer, except you'd be using @op instead of @solid. In general this pattern is called an <https://docs.dagster.io/concepts/ops-jobs-graphs/ops#op-factory|op factory>