https://dagster.io/ logo
#ask-community
Title
# ask-community
s

sar

08/11/2022, 5:56 PM
I can’t seem to figure out how to make the following call through dagster:
dbt deps
I am looking here (https://docs.dagster.io/_apidocs/libraries/dagster-dbt#cli-resources) but don’t seem to understand how to make the call using
dbt_cli_resource
c

claire

08/11/2022, 9:41 PM
Hi Sar, something you can do here is create a op that requires the
dbt_cli_resource
and then makes a
cli
call via the resource. For example:
Copy code
@op(required_resource_keys={"dbt_cli"})
def dbt_deps_op(context):
    context.resources.dbt_cli.cli("deps")
We'll likely also add a pre-built
dbt_deps_op
that runs this command soon
s

sar

08/12/2022, 12:02 AM
No luck so far. It’s throwing the following error now:
Copy code
Error loading repository location airbyte.py:dagster._core.errors.DagsterInvalidDefinitionError: resource with key 'dbt_cli' required by op 'dbt_deps_op' was not provided. Please provide a <class 'dagster._core.definitions.resource_definition.ResourceDefinition'> to key 'dbt_cli', or change the required key to one of the following keys which points to an <class 'dagster._core.definitions.resource_definition.ResourceDefinition'>: ['io_manager', 'dbt']
This is the very basic script that i’m trying out:
Copy code
from dagster import job, op
from dagster_dbt import dbt_cli_resource, load_assets_from_dbt_project, dbt_run_op
from dagster._utils import file_relative_path

DBT_PROJECT_DIR = file_relative_path(__file__, "/.dbt/")
DBT_PROFILES_DIR = file_relative_path(__file__, "/.dbt/")

@op
def run_dbt():
    dbt_assets = load_assets_from_dbt_project(
        DBT_PROJECT_DIR,
        DBT_PROFILES_DIR,
    )
    dbt_run_op.alias(name="load_dbt_assets")


@op(required_resource_keys={"dbt_cli"})
def dbt_deps_op(context):
    context.resources.dbt_cli.cli("deps")

@job(resource_defs={
    "dbt": dbt_cli_resource.configured({"project_dir": DBT_PROJECT_DIR, "profiles_dir": DBT_PROJECT_DIR})
    })
def airbyte_job():
    dbt_deps_op()
    dbt_output = run_dbt()
c

claire

08/12/2022, 12:20 AM
Ah, you'll need to switch the code to be:
Copy code
@op(required_resource_keys={"dbt"})
def dbt_deps_op(context):
    context.resources.dbt.cli("deps")
to match the resource key you provided in the job
s

sar

08/12/2022, 1:44 AM
Changing it throws this error:
Copy code
UserWarning: Error loading repository location airbyte.py:dagster._core.errors.DagsterInvalidConfigDefinitionError: Error defining config. Original value passed: 'dbt_cli'. 'dbt_cli' cannot be resolved.
Nevermind, this one was my bad.
g

geoHeil

08/22/2022, 7:48 PM
c

claire

08/22/2022, 8:50 PM
I don't know personally because I haven't been working on the dbt integration, but Owen will know. He's on PTO until Thursday so I'd ask again in #dagster-support then.
a

Adam Bloom

08/22/2022, 11:03 PM
@claire I commented on another thread - but what is the use case where you'd want to run dbt deps as an op? The use case doesn't make sense to me. I'd expect dbt deps to be run once as part of environment setup (so, your user code build). Obviously, there's no requirement to use a new op, but I'm concerned that adding this op opens the door to lots of corner cases.
k

Konrad Schlatte

09/12/2022, 11:17 AM
@Adam Bloom Just fyi - we are running dbt_deps as part of the production build and also CI, in case a developer adds a new dependency. So this saves us running it manually on production or ci.
a

Adam Bloom

09/12/2022, 3:40 PM
@Konrad Schlatte yup, that's exactly when I'd expect it to be run.
k

Konrad Schlatte

09/23/2022, 4:19 PM
sorry @Adam Bloom don't quite understand why you don't think there's a case of running dbt deps as an op. What would you say is the best way of implement it in dagster as part of the build process - i.e.
dbt deps
dbt seed
and
dbt run
?
8 Views