How does the "Source Data" information get populated in the Dagster asset UI

In the Dagster asset UI, the "Source Data" information is populated based on the metadata provided in the dbt project configuration and the relationships defined between assets. When you define a dbt source in your

sources.yml

file and use it in a dbt model, Dagster parses this information and infers the correspondence between the dbt source and the Dagster asset. This allows the UI to show the asset lineage, including the source assets. Here's a step-by-step explanation of how this works: 1. Define dbt Sources: In your dbt project, you define sources in the

sources.yml

file. You can include metadata in this file that specifies the corresponding Dagster asset for each dbt source using the

meta

property. 2. Integrate dbt with Dagster: When integrating dbt with Dagster, you ensure that Dagster reads the contents of the dbt project, including the metadata specified in the

sources.yml

file. 3. Asset Lineage in UI: Dagster's UI can then represent the asset lineage, showing the dependencies between assets. If a dbt model depends on a dbt source, the UI will reflect this relationship. 4. Materialization and Source Data: When assets are materialized, the UI displays information about the source data that the asset consumed. This includes timestamps and other relevant metadata that help users understand when and how the data was processed. 5. Observable Source Assets: For observable source assets, Dagster assumes that these correspond to input data with discrete moments of change. The UI represents this by showing that the downstream asset contains all the data available up to the moment just before a new version of the source data was observed. It's important to note that there have been discussions and issues raised regarding the behavior of how source data is displayed in the UI, especially concerning observable source assets. For example, there have been cases where the UI displayed the timestamp for source data as after the materialization timestamp, which can be confusing for users. These issues are being addressed by the Dagster team to ensure that the UI behavior is intuitive and accurate. For more detailed information on how to define dbt sources and integrate them with Dagster, you can refer to the documentation on using dbt with Dagster and the dagster-dbt integration reference.