The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hey team, I want to use the `load_from_manifest` function to load my dbt assets. However, I don’t commit dbt’s target file to my github repo. How can I use s3 to load and update my manifest.json file through dagster’s dbt runs?

Hey!

Assuming you can get the `manifest.json` into S3 somehow (ex CI/CD):

In your `__init__.py`, make a raw fetch from S3 to get the `manifest.json` and save it to a temp file in the Dagster instance's file system.


Then, you'd have the `manifest.json` accessible to your dagster instance, so you can now read it into Dagster and pass it into `load_assets_from_dbt_manifest`.


You wouldn't use a resource or op for this because this should happen when things are defined.

Let me write out some pseudocode really quickly

And how do I get dagster to overwrite that s3 file with the newly generated manifest.json after a dbt run?

What's your use case where you'd like to update the `manifest.json` after runs?

Usually, you'd only need to update the `manifest.json` whenever dbt models change. In which case, I've seen users have CD for dbt look like this:
• merge PR
• generate `manifest.json` and write to S3
• tell Dagster the dbt project changed and have it refresh its definitions. sometimes done through the GraphQL API to trigger the `reloadWorkspace` mutation. 
    ◦ If you don't need super tight robustness, tbh you can just do it from the UI.

```from dagster import Definitions

from dagster_dbt import load_assets_from_dbt_manifest

import json
import boto3

MANIFEST_PATH = '/tmp/manifest.json'

s3 = boto3.client('s3')
s3.download_file('dbt-artifacts', 'manifest.json', MANIFEST_PATH)

with open(MANIFEST_PATH) as f:
    manifest = json.load(f)

dbt_assets = load_assets_from_dbt_manifest(
    manifest,
    key_prefix="best_target",
)

defs = Definitions(
    assets=dbt_assets,
)```
Something like this. But after typing that out, you probably don't even need to download the file from S3 completely and instead just grab it directly into memory.

Hey <@U03AJ8UAPDW>!
&gt; tell Dagster the dbt project changed and have it refresh its definitions. sometimes done through the GraphQL API to trigger the `reloadWorkspace` mutation
Do you have any examples on how to do this?

lol just saw your Q in <#C01U954MEER|dagster-support> and started drafting up an example

awesome, thanks! sorry for posting in two places, wasn’t sure where would be most appropriate