I can’t seem to figure out how to make the following call through dagster:

dbt deps

I am looking here (https://docs.dagster.io/_apidocs/libraries/dagster-dbt#cli-resources) but don’t seem to understand how to make the call using

dbt_cli_resource

Hi Sar, something you can do here is create a op that requires the

dbt_cli_resource

and then makes a

cli

call via the resource. For example:

@op(required_resource_keys={"dbt_cli"})
def dbt_deps_op(context):
    context.resources.dbt_cli.cli("deps")

No luck so far. It’s throwing the following error now:

Error loading repository location airbyte.py:dagster._core.errors.DagsterInvalidDefinitionError: resource with key 'dbt_cli' required by op 'dbt_deps_op' was not provided. Please provide a <class 'dagster._core.definitions.resource_definition.ResourceDefinition'> to key 'dbt_cli', or change the required key to one of the following keys which points to an <class 'dagster._core.definitions.resource_definition.ResourceDefinition'>: ['io_manager', 'dbt']

This is the very basic script that i’m trying out:

from dagster import job, op
from dagster_dbt import dbt_cli_resource, load_assets_from_dbt_project, dbt_run_op
from dagster._utils import file_relative_path

DBT_PROJECT_DIR = file_relative_path(__file__, "/.dbt/")
DBT_PROFILES_DIR = file_relative_path(__file__, "/.dbt/")

@op
def run_dbt():
    dbt_assets = load_assets_from_dbt_project(
        DBT_PROJECT_DIR,
        DBT_PROFILES_DIR,
    )
    dbt_run_op.alias(name="load_dbt_assets")


@op(required_resource_keys={"dbt_cli"})
def dbt_deps_op(context):
    context.resources.dbt_cli.cli("deps")

@job(resource_defs={
    "dbt": dbt_cli_resource.configured({"project_dir": DBT_PROJECT_DIR, "profiles_dir": DBT_PROJECT_DIR})
    })
def airbyte_job():
    dbt_deps_op()
    dbt_output = run_dbt()

Ah, you'll need to switch the code to be:

@op(required_resource_keys={"dbt"})
def dbt_deps_op(context):
    context.resources.dbt.cli("deps")

to match the resource key you provided in the job

Changing it throws this error:

UserWarning: Error loading repository location airbyte.py:dagster._core.errors.DagsterInvalidConfigDefinitionError: Error defining config. Original value passed: 'dbt_cli'. 'dbt_cli' cannot be resolved.

@claire https://dagster.slack.com/archives/C01U954MEER/p1660254343843889?thread_ts=1660240572.201999&cid=C01U954MEER do you have any concrete date when soon will arrive in a release?

I don't know personally because I haven't been working on the dbt integration, but Owen will know. He's on PTO until Thursday so I'd ask again in #dagster-support then.

@claire I commented on another thread - but what is the use case where you'd want to run dbt deps as an op? The use case doesn't make sense to me. I'd expect dbt deps to be run once as part of environment setup (so, your user code build). Obviously, there's no requirement to use a new op, but I'm concerned that adding this op opens the door to lots of corner cases.

@Adam Bloom Just fyi - we are running dbt_deps as part of the production build and also CI, in case a developer adds a new dependency. So this saves us running it manually on production or ci.

@Konrad Schlatte yup, that's exactly when I'd expect it to be run.

sorry @Adam Bloom don't quite understand why you don't think there's a case of running dbt deps as an op. What would you say is the best way of implement it in dagster as part of the build process - i.e.

dbt deps

dbt seed

and

dbt run

?