https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
f

Filipe Almeida

02/13/2023, 2:17 PM
Hi I am playing around with the
load_assets_from_dbt_project
and I notice can load seeds as an asset if I set
use_build_command=True
.
Copy code
use_build_command (bool): Flag indicating if you want to use `dbt build` as the core computation
            for this asset, rather than `dbt run`.
Could you tell me a potential problem the disadvantage of trying to load seeds as assets like this? Will it be slower for example?
s

Stephen Bailey

02/13/2023, 4:13 PM
One potential problem is that to materialize assets this way, you will always run `dbt build --select my_asset.``` t . The build command runs
seed
,
run
,
snapshot
, and
test
(I think). The biggest issue may bethat you don't want to run
test
every time you materialize an asset.
👍 2
o

owen

02/13/2023, 11:02 PM
The above is a great summary of the situation -- theoretically, I think you could pass in an
exclude="resource_type:test"
to
load_assets_from...
to avoid that sort of thing but I'm not 100% sure on that. Seeds specifically are also interesting as it's often "wasteful" to run seeds after the first time they've been materialized (if they don't change). If your seeds are pretty small then it's probably not worth worrying about but I can imagine situations where that extra time starts adding up. We are working on automatically "versioning" the dbt asset definitions, so we could in theory detect that a dbt seed hasn't changed and then make it easier to skip running seeds which have not changed since their last materialization, but this would be a bit further out.
f

Filipe Almeida

02/14/2023, 9:47 AM
Thank you so much for this! 🤩
n

Nicolas Parot Alvarez

04/07/2023, 12:25 PM
Hey @owen, is there an issue we can track somewhere to know when the seed update detection could happen? Currently, we're refreshing seeds on a daily schedule. The seeds may change a couple of times a year, but we want it to be updated on D+1 automatically if a change happens. So, we would also be interested in having a smarter solution to only run a seed when the source file has been updated.