19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#announcements
Title
# announcements
b
Basil V
10/14/2020, 6:13 PMAre there any examples available for using
dagster-dbtd
dwall
10/14/2020, 6:14 PMhey @Basil V 👋 we haven't made any examples public, but I'm more than happy to send over a few examples of how we use it internally
b
Basil V
10/14/2020, 6:15 PMThat would be awesome! Feel free to DM them or whatever you prefer. Thanks!
👍 1
d
dwall
10/14/2020, 6:21 PMso we run dbt as an RPC server and dagster triggers production dbt processes by communicating with the server. In practice, our pipelines tend to look something like the following:
most of the time, we are triggering some form of data ingestion and/or transformation to happen pre-dbt, and then triggering a number of dbt processes to happen after (sometimes in parallel, as shown above)
we surface all dbt flags via the Dagster UI so that end users can dynamically change the configuration of run as well:
additionally, we surface all dbt logs through the Dagster UI:
b
Basil V
10/14/2020, 6:24 PMThanks for the info. Still trying to wrap my head around it I think. Previously I was just able to plug in my dbt model and run it as a solid. Is there some way to do the equivalent in this new setup?
d
dwall
10/14/2020, 6:25 PMwe only really use the RPC solids so I can only speak to that, but we have a single solid
dbt_rpc_run_and_waitso the "plugging in" of your dbt model would be equivalent to invoking the
dbt_rpc_run_and_waitb
Basil V
10/14/2020, 6:26 PM Copy code
from dagster_dbt import create_dbt_run_solid
dbt_parse_send_logs = create_dbt_run_solid(
project_dir='path/to/my_dbt_model/',
name='dbt_model_name',
)This is basically what I was doing before
d
dwall
10/14/2020, 6:26 PMoh yeah the way I mentioned above is a more dynamic way of doing what you're doing
the model selection is just configuration on the solid itself
b
Basil V
10/14/2020, 6:26 PMSo do I have to set up and have dbt running as a separate server now to do the RPC?
d
dwall
10/14/2020, 6:27 PMyeah, RPC requires dbt to be running as a server
but thats just one way to run dbt via Dagster
there is also a
cli(I'm less familiar with that one)
interface looks similar though: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cli/solids.py#L96-L157
b
Basil V
10/14/2020, 6:29 PMOk I'll check these resources out and will circle back with you if I still end up stuck. Thanks again!
👍 1
b
bob
10/14/2020, 6:55 PM@Basil V You are right that
dagster-dbtdagster-dbtb
Basil V
10/14/2020, 6:57 PMThanks!
👍 1
b
bob
10/14/2020, 6:59 PMas @dwall mentioned, there are solids for the both dbt RPC server and the dbt CLI.
in the case of dbt CLI solids, you would pass in CLI options via the solid configs, such as
--project-dir--models--exclude2 Views