https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
i

Ivan Tsarev

05/17/2023, 12:38 PM
Hi folks! Do I get it correct, there is no way to pass any additional run argument to
dagster_dbt.dbt_cloud_run_op
? My case is kinda basic, I have two environment for dagster and one dbt Cloud project. Dbt manages prod/dev DWH separation by itself by target mechanism - when it is defined as "prod" in Jobs it will use prod DWH, in any other cases it will fall to dev DWH. So we do have target set as "prod" in most of our jobs to use it as prod way to work with DWH. However, we are now switching to dagster for orchestration of dbt and for sure we want more or less same functionality with prod/test. And I figured out 2 and a half workout right now: • Have two different sets of Jobs in dbt. One to be started from prod Dagster and another from dev. To be honest it sounds awful and is terrible in terms of scalability with number of dbt Jobs. • Defining our own ops based on
DbtCloudClient
but I not quite sure how stable this api is especially with this comment in source code:
Copy code
# TODO: This resource should be a wrapper over an existing client for a accessing dbt Cloud,
# rather than using requests to the API directly.
• Making PR to dagster adding this functionality which is probably ideal but definitely not as quick as I want :) So any other ideas?
q

Qwame

05/17/2023, 2:27 PM
I believe the
DbtCloudClient
accepts all arguments that can be passed when using the Dbt API.
dbt.get_dbt_client().run_job_and_poll
should give you all that. You can pass
target_override
, etc. and all arguments you can pass to the dbt cloud API here
i

Ivan Tsarev

05/17/2023, 2:34 PM
@Qwame yep, that's my second idea in the list, if I get you right. But as I pointed, this class is part of dagster "guts" and can probably be changed with breaking my new ops/resource based on it. Especially with mentioned TODO in its code 🙂
q

Qwame

05/17/2023, 2:47 PM
What version of dagster are you on @Ivan Tsarev
i

Ivan Tsarev

05/17/2023, 2:48 PM
@Qwame ~=1.3
q

Qwame

05/17/2023, 3:14 PM
Yeah, I believe the team is working on improving the DBT integration, even though it's one of the best out there AFAIK. However, you can use the
run_job_and_poll
for your needs and later migrate if these changes are integrated.
2 Views