Flavien
08/10/2022, 6:56 AMdagster-dbt
integration and specifically load_assets_from_dbt_manifest
. This function defines two parameters:
• `select`: a dbt selection string for the models in a project that you want to include.
• `selected_unique_ids`: the set of dbt unique_ids that you want to load as assets.
They appear quite similar so I guessed that select
limits what gets executed by dbt run
while selected_unique_ids
selects what gets exposed as an asset in Dagster. Is it correct?
I wanted to test that assumption so I created this simple DAG in dbt a --> b
and loaded the asset this way:
assets = load_assets_from_dbt_manifest(manifest, select="+b", selected_unique_ids={"model.dbt_project.b"})
but I got the following error:
dagster._core.errors.DagsterInvariantViolationError: Core compute for op "run_dbt_dbt_project_aeba1" returned an output "a" that does not exist. The available outputs are ['b']
After digging into the integration code, I saw that the op does not account for the selected_unique_ids
and yield materialization for every dbt node. Bug?
An other related question is: why is the selected_unique_ids
parameter not present in the definition of load_assets_from_dbt_project
?owen
08/10/2022, 4:55 PMselected_unique_ids
parameter is generally is not useful to set yourself (and the docs should be updated to reflect this). basically, it represents the "resolved" unique ids from a dbt select statement, and it should either be None
, or exactly match the set of unique ids specified by the select
parameter. It mostly exists for legacy reasons -- originally load_assets_from_dbt_manifest
did not support the select
parameter (for various reasons), so this was an alternate way for specifying that selection. Nowadays, the select parameter is generally a much more convenient way of specifying that information.Flavien
08/12/2022, 8:36 AMexclude
parameter. I saw you created an issue last month to do just that. Are you open to contributions?owen
08/12/2022, 4:11 PM