https://dagster.io/ logo
Powered by
#dagster-support
Title
# dagster-support
w

wp

08/19/2022, 3:08 PM
still running into issue with a dbt run-operation op not returning an output using
Copy code
dbt_result = context.resources.dbt.run_operation(
        "my_macro",
        args={"type": 'value'}
    )
log is showing 
Yielded output "result" of type "Any". (Type check passed).
instead of a 
DbtCliOutput
as indicated from the documentation. Trying to log dbt_result is showing {} too. I know the macro ran because of log statement of the return results. Any ideas what I might be missing? For more contex, my_macro just returns a count so the next op can evaluate if it should run based on the count. Edit2: yup. I guess 
DbtCliOutput
will only work with run_results.json so only dbt command that produces it will have a 
DbtCliOutput
anyone have different approach to using a dbt output(for example a count of table) for a downstream op?
a

Adam Bloom

08/19/2022, 4:08 PM
are you setting the output type in your 
op
definition? won't help with the result being empty, but would make the log more clear:
Copy code
@op(
out=Out(DbtOutput, description="Parsed output from running the dbt command."),
)
o

owen

08/19/2022, 4:12 PM
hi @wp! A 
DbtCliOutput
should be returned regardless of if there's a run_results.json file (it's just that the 
run_results
field on the object will be None if that file doesn't exist), so I think something else might be going on. Do you mind sharing the full op definition that you're using? Adam's suggestion also makes sense to try
w

wp

08/19/2022, 4:19 PM
Copy code
@op(
    out=Out(DbtOutput, description="Parsed output from running the dbt command."),
    required_resource_keys={"dbt"}
)
def drop_rollup_full_refresh(context):
    context.resources.dbt.cli("deps")
    tag = context.op_config["tag"]
    source_id = context.op_config["source_id"]
    workspace_id = context.op_config["workspace_id"]

    dataset = f"ws_{workspace_id}"
    project = os.getenv("GCP_PROJECT")

    os.environ["DBT_DATASET"] = dataset
    os.environ["DBT_PROJECT"] = project

    # check sourceType
    dbt_result = context.resources.dbt.run_operation(
        "source_type_check",
        args={"sourceType": "GOOGLE_ADS"}
    )

    <http://logger.info|logger.info>(f"Response1: {dbt_result}")
    <http://logger.info|logger.info>(f"Response2: {dbt_result.result}")

    #full-refresh rollup
    context.resources.dbt.cli(
        "run --full-refresh",
        models=[f"tag:{tag}"],
        vars={"sourceTable": f"{project}.{dataset}.__TABLES__"}
    )
this is the op. essentially I am trying to run dbt macro to return a table count before determining if the next dbt run (full-refresh rollup) should run
o

owen

08/19/2022, 4:21 PM
hm and you're seeing 
Response1: {}
in the logs?
w

wp

08/19/2022, 4:23 PM
yea 
dbt_result.result
was just 
{}
and this is after adding the output in the op definition as suggested by @Adam Bloom
o

owen

08/19/2022, 4:25 PM
ah I see ok -- yeah originally I assumed that you were trying to return the DbtCliOutput created from 
run_operation
from the op (rather than use that value within the op)
this all seems to line up -- you're getting a DbtCliOutput object, but its 
result
field is None because there's no run_results.json. It might be a bit of a pain, but that DbtCliOutput object has two other fields, 
logs
and 
raw_output
, which should contain the log output from running that dbt command (and it sounds like the value you're looking for could be parsed out of those logs)
as for the output, do you need to pass anything from this op to downstream steps? If not, you can just remove the out parameter from the decorator
w

wp

08/19/2022, 4:32 PM
I guess I shouldn’t have assume that the op can return a query result 😅 because that not dbt’s purpose either. I think i’m leaning on adding a big query op for the purpose instead. Thanks for the clarification!
o

owen

08/19/2022, 4:33 PM
makes sense!
❤️ 1
3 Views
Powered by