https://dagster.io/ logo
d

Daniel Upton (Dexcom)

12/26/2020, 7:43 PM
Hi. I enjoyed Nick Shrock's presentation at Coalesce and on Data Engineering podcast awhile back, and I have a noob question. If I intend to use FiveTran to replicate Oracle data into Snowflake, then use dbt (maybe just the open-source version) for transformations in Snowflake, and I want DAG orchestration including dbt models and beyond the dbt models (FiveTran upstream and, for example, Tableau Data Extracts downstream), and I want to use Dagster for end-to-end orchestration, to what extent to I need dbt-cloud for the dbt-level orchestrations? If I don't use dbt-cloud, what add'l skills does my team need to orchestrate w/ Dagster that we would otherwise not need if using dbt-cloud plus Dagster?
s

schrockn

12/28/2020, 8:51 PM
Hey Thanks Daniel! You certainly don’t need to use dbt cloud in order to use Dagster. You can invoke dbt through the cli runner or the rpc server. In terms of your second question about skills, I’m having a bit of trouble parsing it. However there is a core problem in that our current dbt integration does not have out-of-the-box support for dbt cloud. I’m not sure how different the cloud API is from the rpc API, but I assume it wouldn’t be that different. @dwall definitely has the more expertise than me on these matters. and @bob has a lot of context on the state of the integration
4 Views