Hi, I am trying the Dagster with dbt integration. ...
# ask-community
g
Hi, I am trying the Dagster with dbt integration. I have a config file where i am passing the project_dir and the profile_dir that is required for the DBT models to run. When I try to run the DBT models i keep getting the error as dagster.core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "op_run_dbt": and the associated exception is The above exception was caused by the following exception: AttributeError: 'function' object has no attribute 'run' I am confused - How do i convert a resource definition to a resource ? Example - dagster.core.definitions.resource_definition.ResourceDefinition (This is the object type I am getting in Dagster UI) to dagster_dbt.cli.resources.DbtCliResource' (This is the object type that I require to run models in DBT) Please let me know if any additional information is required
s
@owen
o
hi @Ghanashyam Alakke -- would you mind sharing how you're constructing your job? Dagster will automatically convert resource definitions into resources when a job is invoked, so if you have a job defined like this: https://docs.dagster.io/integrations/dbt#using-dbt_cli_resource-to-run-your-entire-dbt-project, then when you run this job (through dagit or otherwise), the dbt_run_op should get an instance of DbtCliResource passed into it.
g
Thank you for your reply. I will take a look at the documentation that you have recommended and try to see if it will fix the problem. The job is created by a program that my manager wrote. I will check with him if I can share the same with you
Hi - I have attached a sample config file and a sample op file. I am trying to define the dbt_cli_resource at the op level and pass the dbt_profiles and dbt_project as a parameter from the config file. I understand that the dagster job will create the resource; but if i try to create the dbt_cli_resource at the op level, i get the resourcedefinition. Please let me know if we can create the dbt_cli_resource in the op and not from the job. Thank you
o
@Ghanashyam Alakke if you want to set
project_dir
and
profiles_dir
at the op level, the best way is still to use resources at a job level. That would look something like this:
Copy code
@op(required_resource_keys={'dbt'})
def op_run_dbt(context,project_dir:str,profiles_dir:str):
    context.resources.dbt.run(
        project_dir=project_dir,
        profiles_dir=profiles_dir
    )

@job(resource_defs={"dbt": dbt_cli_resource})
def my_job():
    # ...
but if you have a hard requirement of not going through the resource system, you can create a resource instance from a resource definition by invoking it with a value for the "context" parameter (generally None is fine here).
Copy code
@op
def op_run_dbt(context,project_dir:str,profiles_dir:str):
    dbtcli_obj = dbt_cli_resource
    result = dbt_cli_resource.configured(
        {
            "project_dir": project_dir,
            "profiles_dir": profiles_dir,
        }
    )(context=None)
    result.run()
To be clear, this isn't really a recommended pattern, but it should do what you're looking for
g
Thank you for your response Owen. I will try this one out. I will let my manager know about your recommendation. I will update you. Thank you again