I am doing some research on Dagster and how it could be coupled with dbt. I like the way that metadata is captured all in one place, but I have a question about the dbt execution. It seems like dagster is still executing dbt run, so if I have a DAG with 100 models and there is a problem at model 50, it doesn’t seem like there is a way to restart from there after a fix vs starting back at model 1. Is that correct?

Hello. You can find a blog post on dagster with an exemple to play with https://github.com/dagster-io/dagster/tree/master/examples/dbt_example https://dagster.io/blog/dagster-dbt

oh yes, I read this, but there is just a single node for the dbt run. so I get the impression that if somethign faisl part of the way through, you will need to rerun all models

Noel you’re correct that that is our current approach. This is a really interesting area of discussion if you want to discuss it further. There is a lot of potential in projecting dbt model dags into dagster space in order to do finer-grained orchestration. @sandy has prototyped such a projection (as well as a community member if memory serves) but we don’t have any near-term plans to implement. Dagster has a unique opportunity to do something interesting here as we have separate logical and execution graphs. I’d love to be able to render the live progress of a series of dbt executions in our Gantt chart and log viewer.

@schrockn sure, we can discuss further

Just stumbling upon this thread while looking for the same thing as @Noel Gomez. Is there another thread elaborating about the potential approach of regenerating the dbt DAG in dagster?

Hey @Benoit Perigaud. Please reach out to @sandy who has been exploring this area. We’re actively exploring “dbt-native orchestration” and would love to chat with users about it.