https://dagster.io/ logo
Title
e

Emily

07/28/2021, 1:01 PM
wanted to give some feedback on the new dbt resource. for one, it has definitely improved our dbt pipeline and made it much cleaner overall! but one small thing that i came across i think could improve it a little more if possible. when i was initially setting it up, i wanted to set the config like this, where we can set
models
only one time. but in reality, when we try running our pipeline this way we get errors on certain steps (like
run-operation
) since macros dont take
models
as an argument. so in practice, i had to take
models
out of the config here and pass it separately in the
run
and
test
solids. it would be awesome (if possible) to be able to pass
models
in the resource config so we only have to set it once, but only have it pass into the dbt commands where its expected as an input (like
run
and
test
but not
run-operation
)
"resources": {
        "dbt": {
            "config": {
                "target": DBT_TARGET,
                "profiles_dir": DBT_DIR,
                "project_dir": DBT_DIR,
                "models": ["*"],
                "ignore_handled_error": True,
            }
        },
    },
o

owen

07/28/2021, 1:41 PM
hey @Emily — glad the resource has helped! this is great feedback. this behavior came about because there are cases where dbt will add cli flags between versions, and so it’s not always feasible for dagster to know the full set of possible flags for a given command (so it’s nice to be a bit less strict for these cases). however, I think in practice your use case is a very normal/reasonable one, and it deserves a little magic to make it work cleanly. it seems unlikely that dbt will suddenly decide that run-operation needs a models parameter, after all 😛. I’ll see if I can get a fix out for tomorrow’s release!
e

Emily

07/28/2021, 1:44 PM
amazing! thanks so much 🙂
o

owen

07/28/2021, 5:54 PM
a change will be in tomorrow's release that should solve this issue. basically, the
models
,
select
, and
exclude
flags will only be passed from the default flags for commands that use them (and ignored for the others) 😄
e

Emily

07/28/2021, 5:55 PM
woohoo! thanks for the quick turnaround :dagstar:
😛artydagster: 1