Hi, I am currently a data engineer and a beginner with dagster. Our team uses dbt cloud and we wanted to leverage dagster to get insights about our pipeline. I was able to run this example https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/load-dbt-models since it explained which files to change and run. When i wanted to try the dbt cloud tutorial, i could not follow it https://docs.dagster.io/integrations/dbt-cloud as i was not able to understand which file to create and which file has to be run. So wanted to check if there are any workbook/instructions for end to end example of running dagster and dbt cloud? Thank you!

We don't have an end-to-end example for dbt Cloud, however all of the code snippets can be put in your root-level

__init__.py

that has the

Definitions

call in it.

Thanks for your reply.

from dagster_dbt import DbtCloudClientResource
from dagster import EnvVar

dbt_cloud_instance = DbtCloudClientResource(
    auth_token="random",
    account_id="random",
)
from dagster_dbt import load_assets_from_dbt_cloud_job

# Use the dbt_cloud_instance resource we defined in Step 1, and the job_id from Prerequisites
dbt_cloud_assets = load_assets_from_dbt_cloud_job(
    dbt_cloud=dbt_cloud_instance,
    job_id=random,
)

from dagster import (
    ScheduleDefinition,
    define_asset_job,
    AssetSelection,
    Definitions,
)

# Materialize all assets
run_everything_job = define_asset_job("run_everything_job", AssetSelection.all())

# Materialize only the staging assets
run_staging_job = define_asset_job(
    "run_staging_job", AssetSelection.groups("staging")
)

defs = Definitions(
    # Use the dbt_cloud_assets defined in Step 2
    assets=[dbt_cloud_assets],
    schedules=[
        ScheduleDefinition(
            job=run_everything_job,
            cron_schedule="@daily",
        ),
        ScheduleDefinition(
            job=run_staging_job,
            cron_schedule="@hourly",
        ),
    ],
)

I created this python file and ran it and it works but does not do anything. Like i am able to get authorization to work but how do i see these definition in dagster (not sure how to bring these in).

how are you running the file? are you using

dagster dev

locally? in dagster cloud?

@jamie i ran the above code locally. using just python test_file.py (i am trying to use dagster locally along with dbt cloud) and don't have instructions on how to do so.

ok - if you run

dagster dev

and then navigate to the localhost url it prints out (should be

<http://127.0.0.1:3000/>

) you’ll get to the dagster UI If you haven’t seen it yet, I recommend checking out the tutorial, specifically the section on navigating the UI . We also have a dbt tutorial, but it uses dbt core, the general concepts may still be of interest though

Thanks @jamie. I did run this tutorial. The example of jaffle shop project works and i can see those in dagster dev as you indicated. I have a separate dbt cloud project which has a lot of sql models. What i need to know how do i see my dbt cloud assests in dagster (in localhost)?

ok - so for dbt cloud, you can put the code (specifically steps 1 and 2 in this guide) in whatever file (or a new file) you want in your dagster project. Then somewhere in your dagster project there should be a

Definitions

object. Depending on how you set up that Definitions object you may need to add the dbt cloud assets to it. If you find that object and share the code here i can tell you what to do

Hi @jamie i did step 1 and step 2, step 1 connects to the dbt_cloud_instance with the token. and then step 2 creates dbt_cloud_assets.

dbt_cloud_assets = load_assets_from_dbt_cloud_job(
    dbt_cloud=dbt_cloud_instance,
    job_id=290929,
)

i believe by object you mean dbt_cloud_assets with job_id. What to do next after this to see it in dagster ?

so there should be a

Definitions

object somewhere in your code that you’ll want to add

dbt_cloud_assets

to

yes.

from dagster import (
    ScheduleDefinition,
    define_asset_job,
    AssetSelection,
    Definitions,
)

# Materialize all assets
run_everything_job = define_asset_job("run_everything_job", AssetSelection.all())

# Materialize only the staging assets
run_staging_job = define_asset_job(
    "run_staging_job", AssetSelection.groups("staging")
)

defs = Definitions(
    # Use the dbt_cloud_assets defined in Step 2
    assets=[dbt_cloud_assets],
    schedules=[
        ScheduleDefinition(
            job=run_everything_job,
            cron_schedule="@daily",
        ),
        ScheduleDefinition(
            job=run_staging_job,
            cron_schedule="@hourly",
        ),
    ],
)

added it there.

if you run dagster using

dagster dev

it should show them since they are in the definitions object now

assuming all this code is in a python file named test.py

you might need to do

dagster dev -f test.py

i did this and it started running the job_id inside dbt cloud