https://dagster.io/ logo
Powered by
#dagster-feedback
Title
# dagster-feedback
t

Tom Reilly

07/06/2022, 4:05 PM
Any update for when 
load_assets_from_dbt_project
or 
load_assets_from_dbt_manifest
will support dbt seeds and snapshots?
o

owen

07/06/2022, 4:39 PM
hey @Tom Reilly! would you expect both seeds and snapshots to be represented as assets (rather than sources or something like that)?
t

Tom Reilly

07/06/2022, 5:37 PM
I think so. We are pretty new to dagster and are thinking that asset defined jobs will work best for us because our pipelines span multiple technologies. I noticed seeds and snapshots are not loaded using either of those methods which results in an incomplete lineage. I'm assuming this incomplete lineage may prevent an asset defined job from selecting all necessary components. Feel free to correct me if I'm thinking of this the wrong way
Also, I'm seeing some unexpected behavior when materializing an asset and its downstream assets. Right now I only have dbt assets registered via 
load_assets_from_dbt_project
. When materializing a model and it's downstream assets the dbt build works fine and I see in the logs
Copy code
Finished running 8 table models, 388 tests in 293.41s.
Completed successfully
Done. PASS=396 WARN=0 ERROR=0 SKIP=0 TOTAL=396
but afterwards dagster is performing asset materializations and I get a 
KeyError
at this spot
We have a singular test in our testing directory that depends on one of our sources for validation
So that line I linked doesn't really seem right because the test is dependent on a source and 
node_info = manifest_json["nodes"][upstream_id]
tries to find that source within the "nodes" key of the manifest.json when it should probably look within the "sources" key based on the dbt docs
I guess I'm thinking it should look something like
Copy code
for upstream_id in upstream_unique_ids:
    manifest_key = "sources" if upstream_id.startswith("source.") else "nodes"
    node_info = manifest_json[manifest_key][upstream_id]
    upstream_asset_key = node_info_to_asset_key(node_info)
    yield AssetObservation(asset_key=upstream_asset_key, metadata=metadata)
o

owen

07/06/2022, 8:46 PM
hi @Tom Reilly thanks for that report, I'm going to see if I can sneak a change to that effect into tomorrow's release 🙂 as for the snapshots and seeds bit, that'll be slightly more involved but I created an issue for it here: https://github.com/dagster-io/dagster/issues/8770.
🙏 2
✨ 1
6 Views
Powered by