At the scale our team is at in dbt, ~3500 models, ...
# integration-dbt
p
At the scale our team is at in dbt, ~3500 models, we would benefit greatly if materializations are skipped if another one is ongoing. Would love others feedback on this but here is a feature-request targeting this goal - https://github.com/dagster-io/dagster/issues/15488
4
n
I like the feature request! On our side, we prevent excessive materialization by using a sensor to trigger the DBT materialization, and we check in the sensor that some relevant jobs are not already running. It does feel like doing manually, something that could be handled by Dagster in the background. https://gist.github.com/NicolasPA/854392e22dc1410977cc7ddb8b8605a4
p
This is great, thanks!
👌 1
n
@Nicolas Parot Alvarez great idea to check for running materializations. @Paul Burns please see my comment in github. What are your thoughts about having Automaterialize daemon combining all assets into one single run?
🏃 1