https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
m

Mike Needham

12/08/2022, 10:24 PM
so I was able to get my DBT models into dagster, but how do I also get the seeds into dagster
o

owen

12/08/2022, 10:57 PM
hi @Mike Needham! If you set
use_build_command=True
when calling
load_assets_from_dbt*
then seeds will be loaded alongside models
m

Mike Needham

12/08/2022, 11:08 PM
Perfect, thank you
Is there a way to get macros as dagster assets? Have a post hook macro that would be nice to have as an asset
o

owen

12/12/2022, 5:35 PM
hi @Mike Needham! that functionality doesn't exist out of the box at the moment, but you can always write a custom asset for that sort of behavior! I think it would look something like
Copy code
@asset(required_resource_keys={'dbt'}, non_argument_deps={AssetKey("some_dep")})
def my_macro_asset(context) -> None:
    context.resources.dbt.run_operation("my_macro")
m

Mike Needham

12/12/2022, 8:45 PM
I have no idea where/how to add that. I am just getting going in this and am trying to take our existing dbt project and see if I can use it in dagster
o

owen

12/12/2022, 8:57 PM
if you're interested, here's a tutorial that walks you through combing custom assets with assets loaded from a dbt project: https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster. in this case, your custom asset would just be invoking the run_operation function using the same
dbt
resource that the assets loaded from your dbt project use
m

Mike Needham

12/13/2022, 5:37 PM
will take a look at that. thanks
so I tried to modify this example to my dbt project and must be missing something. the dbt project will work fine if I do not do the Step 2 in the example. If I add that it errors out with Error loading repository location tutorial_dbt_dagsterdagster. core.errors.DagsterImportError Encountered ImportError:
cannot import name 'Definitions' from 'dagster' (/home/test/.local/lib/python3.9/site-packages/dagster/__init__.py)
while importing module tutorial_dbt_dagster. Local modules were resolved using the working directory
/home/test/dagster/PM3/tutorial_template
. If another working directory should be used, please explicitly specify the appropriate path using the
-d
or
--working-directory
for CLI based targets or the
working_directory
configuration option for workspace targets.
o

owen

12/21/2022, 2:19 PM
ah you might need to update dagster to the latest version --
Definitions
is a newer API. You can also take a look at an older version of those docs (dropdown in the top left of the docs page) which should reference
repository
m

Mike Needham

12/21/2022, 3:21 PM
that would explain it hopefully. thanks
was able to get it loaded, but how do I set it up so that the macro is dependent on one of the dbt models(assets)?
I am not understanding the linkage and have tried a few things and cannot seem to get it to be a dependant of the last model I have in my dbt graph
o

owen

12/21/2022, 9:03 PM
the first step would be to figure out the asset key that's been assigned to the last model of the graph. In most cases, that'll just be AssetKey("some_model_name"), but sometimes it'll be a two part key like AssetKey(["prefix", "some_model_name"]). Once you know that key, you should be able to create your own custom asset:
Copy code
@asset(required_resource_keys={'dbt'}, non_argument_deps={AssetKey("some_model_name")})
def my_macro_asset(context) -> None:
    context.resources.dbt.run_operation("my_macro")
this is pretty similar to what's described here: https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/part-four, but because you're not actually loading the data from the dbt model into your python function, you don't need to worry about the IOManager stuff
m

Mike Needham

12/21/2022, 9:38 PM
can you have more than one dependency?
what I am trying to do is run a dbt macro that is dependent on 5 models that get materialized. I can get a single model dependency to work with @asset(required_resource_keys={'dbt'},non_argument_deps={AssetKey("model1")},key_prefix=["dbtKeys"], group_name="intermediate") but when I try and figure out how to add the other models as dependents I cannot get it to work. Have tried a few variations onAssetKey=(["model","model2"]) with strings and no luck
sorry for all the questions, just having a hard time getting my head around this
o

owen

12/21/2022, 9:53 PM
oh I see -- each
AssetKey
represent a unique asset. so you'd want
non_argument_deps={AssetKey("model1"), AssetKey("model2")}
m

Mike Needham

12/21/2022, 9:55 PM
perfect, let me add that
that is exactly what I was looking for. thank you so much
🌈 1
131 Views