https://dagster.io/ logo
#ask-community
Title
# ask-community
v

Vyom Raj

08/13/2023, 3:18 AM
Hi, I am currently a data engineer and a beginner with dagster. Our team uses dbt cloud and we wanted to leverage dagster to get insights about our pipeline. I was able to run this example https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/load-dbt-models since it explained which files to change and run. When i wanted to try the dbt cloud tutorial, i could not follow it https://docs.dagster.io/integrations/dbt-cloud as i was not able to understand which file to create and which file has to be run. So wanted to check if there are any workbook/instructions for end to end example of running dagster and dbt cloud? Thank you!
t

Tim Castillo

08/14/2023, 11:37 PM
We don't have an end-to-end example for dbt Cloud, however all of the code snippets can be put in your root-level
__init__.py
that has the
Definitions
call in it.
v

Vyom Raj

08/17/2023, 3:07 AM
Thanks for your reply.
Copy code
from dagster_dbt import DbtCloudClientResource
from dagster import EnvVar

dbt_cloud_instance = DbtCloudClientResource(
    auth_token="random",
    account_id="random",
)
from dagster_dbt import load_assets_from_dbt_cloud_job

# Use the dbt_cloud_instance resource we defined in Step 1, and the job_id from Prerequisites
dbt_cloud_assets = load_assets_from_dbt_cloud_job(
    dbt_cloud=dbt_cloud_instance,
    job_id=random,
)

from dagster import (
    ScheduleDefinition,
    define_asset_job,
    AssetSelection,
    Definitions,
)

# Materialize all assets
run_everything_job = define_asset_job("run_everything_job", AssetSelection.all())

# Materialize only the staging assets
run_staging_job = define_asset_job(
    "run_staging_job", AssetSelection.groups("staging")
)

defs = Definitions(
    # Use the dbt_cloud_assets defined in Step 2
    assets=[dbt_cloud_assets],
    schedules=[
        ScheduleDefinition(
            job=run_everything_job,
            cron_schedule="@daily",
        ),
        ScheduleDefinition(
            job=run_staging_job,
            cron_schedule="@hourly",
        ),
    ],
)
I created this python file and ran it and it works but does not do anything. Like i am able to get authorization to work but how do i see these definition in dagster (not sure how to bring these in).
j

jamie

08/22/2023, 2:24 PM
how are you running the file? are you using
dagster dev
locally? in dagster cloud?
v

Vyom Raj

08/22/2023, 2:31 PM
@jamie i ran the above code locally. using just python test_file.py (i am trying to use dagster locally along with dbt cloud) and don't have instructions on how to do so.
j

jamie

08/22/2023, 2:38 PM
ok - if you run
dagster dev
and then navigate to the localhost url it prints out (should be
<http://127.0.0.1:3000/>
) you’ll get to the dagster UI If you haven’t seen it yet, I recommend checking out the tutorial, specifically the section on navigating the UI . We also have a dbt tutorial, but it uses dbt core, the general concepts may still be of interest though
v

Vyom Raj

08/23/2023, 5:10 PM
Thanks @jamie. I did run this tutorial. The example of jaffle shop project works and i can see those in dagster dev as you indicated. I have a separate dbt cloud project which has a lot of sql models. What i need to know how do i see my dbt cloud assests in dagster (in localhost)?
j

jamie

08/23/2023, 5:56 PM
ok - so for dbt cloud, you can put the code (specifically steps 1 and 2 in this guide) in whatever file (or a new file) you want in your dagster project. Then somewhere in your dagster project there should be a
Definitions
object. Depending on how you set up that Definitions object you may need to add the dbt cloud assets to it. If you find that object and share the code here i can tell you what to do
v

Vyom Raj

08/24/2023, 3:03 PM
Hi @jamie i did step 1 and step 2, step 1 connects to the dbt_cloud_instance with the token. and then step 2 creates dbt_cloud_assets.
Copy code
dbt_cloud_assets = load_assets_from_dbt_cloud_job(
    dbt_cloud=dbt_cloud_instance,
    job_id=290929,
)
i believe by object you mean dbt_cloud_assets with job_id. What to do next after this to see it in dagster ?
j

jamie

08/24/2023, 3:15 PM
so there should be a
Definitions
object somewhere in your code that you’ll want to add
dbt_cloud_assets
to
v

Vyom Raj

08/24/2023, 3:15 PM
yes.
Copy code
from dagster import (
    ScheduleDefinition,
    define_asset_job,
    AssetSelection,
    Definitions,
)

# Materialize all assets
run_everything_job = define_asset_job("run_everything_job", AssetSelection.all())

# Materialize only the staging assets
run_staging_job = define_asset_job(
    "run_staging_job", AssetSelection.groups("staging")
)

defs = Definitions(
    # Use the dbt_cloud_assets defined in Step 2
    assets=[dbt_cloud_assets],
    schedules=[
        ScheduleDefinition(
            job=run_everything_job,
            cron_schedule="@daily",
        ),
        ScheduleDefinition(
            job=run_staging_job,
            cron_schedule="@hourly",
        ),
    ],
)
added it there.
Copy code
Definitions
has
Copy code
dbt_cloud_assets
now how to show these definitions in dagster?
j

jamie

08/24/2023, 3:17 PM
if you run dagster using
dagster dev
it should show them since they are in the definitions object now
v

Vyom Raj

08/24/2023, 3:17 PM
assuming all this code is in a python file named test.py
j

jamie

08/24/2023, 3:17 PM
you might need to do
dagster dev -f test.py
v

Vyom Raj

08/24/2023, 3:20 PM
i did this and it started running the job_id inside dbt cloud
3 Views