https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
p

Paul Burns

08/04/2023, 1:28 PM
At the scale our team is at in dbt, ~3500 models, we would benefit greatly if materializations are skipped if another one is ongoing. Would love others feedback on this but here is a feature-request targeting this goal - https://github.com/dagster-io/dagster/issues/15488
4
n

Nicolas Parot Alvarez

08/04/2023, 2:31 PM
I like the feature request! On our side, we prevent excessive materialization by using a sensor to trigger the DBT materialization, and we check in the sensor that some relevant jobs are not already running. It does feel like doing manually, something that could be handled by Dagster in the background. https://gist.github.com/NicolasPA/854392e22dc1410977cc7ddb8b8605a4
p

Paul Burns

08/04/2023, 2:38 PM
This is great, thanks!
👌 1
n

Nikolaj Galak

08/04/2023, 8:18 PM
@Nicolas Parot Alvarez great idea to check for running materializations. @Paul Burns please see my comment in github. What are your thoughts about having Automaterialize daemon combining all assets into one single run?
🏃 1