I’m still finding the docs really convoluted and f...
# dagster-feedback
s
I’m still finding the docs really convoluted and frustrating. I’m trying to set up a dbt-cloud integration and have been following along with this page. A few issues came up: 1.
EnvVar
didn’t work for me. It didn’t seem to substitute the values that the var held. When I changed them to
os.getenv
, it worked fine. So, what are the missing pieces here to get
EnvVar
working in this context? 2. A job called
run_staging_job
is defined that selects only assets in the
"staging"
group. However, it never discusses how to add dbt assets to this group. When I tried it in my project, this job was empty. 3. Step 4 is optional, but it never mentions why you might want to use this option. It wasn’t until I was developing locally and noticed that dagster was spinning up a
dbt compile
job every few minutes that I realized that it’s practically necessary for local development.
l
Hey man I’m on the same journey as you are, I asked the same questions too. Maybe the responses I got from the team will be relevant for you as well. 1. On EnvVar 2. On dbt compile
s
Thanks Lydia. Any luck figuring out how to tell dagster to only run dbt models with certain tags or model paths?
b
For adding to the group,
load_assets_from_dbt_project
accepts
node_info_to_group_fn
which you can use with a simple lambda
node_info_to_group_fn=lambda node_info: "my_group"
to set them all as a static group (Feedback to anyone from dagster team: might be more ergonomic to allow for a static string version of the argument!)
It also includes a
select
argument you can use as you would the dbt command line
select
to import only certain tags (e.g.
tag:my-tag
) with that function, so you can select certain dbt assets to be loaded into one group, and others into another group.
I think dbt suggests using project path rather than tag for this sort of thing, but I've yet to move to that approach.
s
Thanks Brendan. I haven’t looked into it yet, but Owen told me that tag selectors only work with dbt-core. Have you done this with dbt-cloud?
b
Ah - I totally misread that.
That seems a big missing feature. 😕
s
Yeah, starting to think that I might need to finally learn how to use git submodules blob worry
😓 1
s
Hey @Sterling Paramore we were able to get the integration working but theres a bunch of limitations w/ dbt-cloud specifically i have not for the life of me figured out how to tell dagster to just run specific model paths using dbt-cloud without causing a lot of queueing problems.