hi all i m noticing that when a test fails the asset associa dagster #integration-dbt

hi all, i'm noticing that when a test fails, the a...

Leo Qin

05/09/2023, 11:49 PM

hi all, i'm noticing that when a test fails, the asset associated with the test still records a materialization event - is this expected behavior?

rex

05/10/2023, 9:49 PM

Yeah. I am assuming you are running

dbt build

, which essentially runs

dbt run

and

dbt test

together (link). we currently output

AssetMaterialization

and

AssetObservation

events using the run results and provided manifest. What behavior were you expecting here?

Félix Tremblay

05/12/2023, 1:22 PM

@Leo Qin It is dbt's normal behavior. If you are looking for blue/green deploy, it can be done but it needs to be configured. There's a post about it in dbt discourse

Félix Tremblay

05/12/2023, 1:26 PM

@rex do you think dagster-dbt could provide better support to help users implement blue/green deploy? A Tutorial in the docs would be a great start. Ultimately, Dagster could handle much of the logic by itself

rex

05/12/2023, 1:27 PM

I assume that Leo is talking about the materialization event behavior in Dagster. As Félix mentioned, this is standard dbt behavior, that the model is materialized first in the data warehouse, and then tests are run. Even if the tests fail, the model is still materialized. In Dagster, we create an

AssetMaterialization

that corresponds to a dbt model materialization. Currently, the behavior is that this

AssetMaterialization

is at the same time a dbt model is materialized, which is essentially mimicking the behavior you see in dbt.

rex

05/12/2023, 1:31 PM

Yeah we’re looking to support these custom patterns more thoroughly soon. The basis of this support is that we’ll let you generate software defined assets for your dbt models out-of-the-box, but you can also specify the compute function that generates these assets. Think of it as a custom

@multi_asset

decorator, but for the

dagster-dbt

. Once we expose the compute function to y’all, you can do what you want to achieve your use cases e.g. • blue/green deploy, • run arbitrary dbt commands together, like

dbt run

dbt test

, instead of just

dbt run

dbt build

• use dbt together with Slack/S3/arbitrary resources • etc

Félix Tremblay

05/12/2023, 1:59 PM

Hey @rex that sounds like a very powerful feature for advanced use cases. I'm wondering if that might be a bit intimidating for novice users. Do you think that Dagster could include a tutorial for blue/green deploys with dbt?

rex

05/12/2023, 2:05 PM

Yeah I think we want to strike a balance here: 1. We want to empower our users to be able to take full advantage of the frameworks at play here (dagster + dbt) 2. But yes, for canonical and established use cases for dbt (like the blue/green deploy you mentioned), I think we will have tutorials to show how this works in practice with (1).

👌 1

Leo Qin

05/12/2023, 2:27 PM

Oh interesting - yeah, I was investigating it because I was writing some notifications enhancements, but now that you mention it, the adapter we use allows for zero-downtime builds, which i think can be posed as some variation of a blue/green deploy

2 Views

Open in Slack

Previous Next