I am doing some research on Dagster and how it could be coupled with dbt. I like the way that metadata is captured all in one place, but I have a question about the dbt execution. It seems like dagster is still executing dbt run, so if I have a DAG with 100 models and there is a problem at model 50, it doesn’t seem like there is a way to restart from there after a fix vs starting back at model 1. Is that correct?
I guess something like this could be done in Dagster has anyone done so?
Building a Scalable Analytics Architecture with Airflow and dbt: Part 1 (astronomer.io)
oh yes, I read this, but there is just a single node for the dbt run. so I get the impression that if somethign faisl part of the way through, you will need to rerun all models
06/07/2021, 12:53 AM
Noel you’re correct that that is our current approach. This is a really interesting area of discussion if you want to discuss it further. There is a lot of potential in projecting dbt model dags into dagster space in order to do finer-grained orchestration. @sandy has prototyped such a projection (as well as a community member if memory serves) but we don’t have any near-term plans to implement. Dagster has a unique opportunity to do something interesting here as we have separate logical and execution graphs. I’d love to be able to render the live progress of a series of dbt executions in our Gantt chart and log viewer.
06/07/2021, 12:42 PM
@schrockn sure, we can discuss further
10/20/2021, 11:23 AM
Just stumbling upon this thread while looking for the same thing as @Noel Gomez. Is there another thread elaborating about the potential approach of regenerating the dbt DAG in dagster?
10/20/2021, 11:54 AM
Hey @Benoit Perigaud. Please reach out to @sandy who has been exploring this area. We’re actively exploring “dbt-native orchestration” and would love to chat with users about it.