https://dagster.io/ logo
Title
e

Edson Henrique

03/12/2023, 4:08 PM
how to create a dagster's job to run the external-tables macro? the cli command is basically:
dbt run-operation stage_external_sources --args "select: ..."
, how can i execute that with dagster?
from hyprster.dbt_jobs.resource import cfg
from dagster_dbt import load_assets_from_dbt_project
from dagster import ScheduleDefinition, AssetSelection,  define_asset_job, op


@op
def refresh_morning_external_tables(context):
    return context.resources.dbt.run_operation("stage-external-sources --exclude received_campaigns")

dbt_freshness_5am_assets = load_assets_from_dbt_project(
    project_dir = cfg['project_dir'],
    profiles_dir = cfg['profiles_dir'],
    select = "tag:freshness_5am",
    key_prefix = ["freshness_5am"]
)

dbt_freshness_5am_job = define_asset_job(name = "update_dbt_assets_at_5am", selection = AssetSelection.groups("freshness_5am"))

dbt_freshness_5am_job_schedule = ScheduleDefinition(
    name="dbt_freshness_5am_job_schedule",
    job=dbt_freshness_5am_job,
    cron_schedule="0 5 * * *"
)
this is my current code, but i dont`t know how to add the op to the asset job.
j

Julien DEBLANDER

04/24/2023, 3:36 PM
Hi @Edson Henrique, Did you find any solution?
e

Edson Henrique

04/26/2023, 5:34 PM
@Julien DEBLANDER i migrated the external-sources definitions to dagster's assets
:rainbow-daggy: 1
g

geoHeil

05/09/2023, 1:02 PM
@Sean Lopp is this also allowing for the SELECT statement to be pushed down (for the dbt run operation stage_external_sources)? i.e. when selecting an asset in dagster is the required staging command executed as well (automatically)? The @op approach here looks like this must be handled manually. @Julien DEBLANDER did you find a more automated/integrated solution with the assets?
s

Sean Lopp

05/09/2023, 1:15 PM
cc @owen / @rex who are working on making the dbt integration more extensible Today if you did take the op/asset approach, you could use a sensor to connect the two instead of relying on cron schedules being in sync
g

geoHeil

05/09/2023, 1:25 PM
@Sean Lopp do you think you could provide an example? How could the sensor consider the selections? I.e. of only some assets are selected?