https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
g

geoHeil

06/08/2023, 3:56 PM
@rex https://github.com/dagster-io/dagster/discussions/14477#discussioncomment-6124333 I think it almost works even in the dynamic way sufficiently well. The only remaining issue is the layout of the SQL code. Can the formatting be changed to look as nice as in the old API?
r

rex

06/08/2023, 4:00 PM
Did you try
MetadataValue.md
? What did you get?
g

geoHeil

06/08/2023, 4:29 PM
this is not 100% exactly the same - but I think good enough to get the job done
Execution Started At Execution Completed At Execution Duration also are missing - but it should be fairly simple to re-add them. https://github.com/dagster-io/dagster/discussions/14477#discussioncomment-6124650 however: 1) How is
stream
different from
stream_raw_events
in the example linked from the dagster git? 2) Is the
Copy code
event_node_info
the same type of event or should this go outside the loop?
r

rex

06/08/2023, 4:32 PM
just to be clear, these are different APIs you are talking about. • When you are talking about formatting in the old API, you are describing the description of the Software-defined asset. This occur at definition time. • The API you are using is the metadata for the asset materialization. This occurs in a computation at run time.
Execution Started At
Execution Completed At
Execution Duration
Sure, these could be added back as default metadata entries. Curious why you find them valuable?
The original thought process for leaving them out was that we wanted the user to make a conscious decision about what metadata they want to add as part of their asset materializations. As the Dagster framework, we didn’t want to make this assumption on behalf of the user.
👍 1
g

geoHeil

06/08/2023, 4:34 PM
Runtime is the most interesting to be honest
👍🏽 1
But when trying to retrofit these (delbierately) AttributeError: 'Output' object has no attribute 'event' is failing
How can I select the right events?
where can I get hold of the `
Copy code
event
in my function with the list of streamed events?
• You need to stream the raw events using
.stream_raw_events
. These are the raw logs that are outputted by dbt. Their API is as follows: https://docs.getdbt.com/reference/events-logging#structured-logging • Then for each event, you convert them to their corresponding Dagster events (
Output
,
AssetObservation
) • For the `Output`’s, then you look at the original raw log and extract the relevant information from it.
g

geoHeil

06/08/2023, 4:42 PM
Is it guaranteed that all observation events occur before the out? And that all assets occur in order : observation 1 out 1, observation 2 out 2?
r

rex

06/08/2023, 4:43 PM
They occur in the order that is outputted by the dbt logs.
g

geoHeil

06/08/2023, 7:45 PM
Great! This works fine
3 Views