The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Are there any examples available for using `dagster-dbt` ? It looks like the library has changed quite a bit since I used it previously. Tx!

hey <@UUYAH8SFJ> :wave: we haven't made any examples public, but I'm more than happy to send over a few examples of how we use it internally

That would be awesome! Feel free to DM them or whatever you prefer. Thanks!

Screen Shot 2020-10-14 at 11.19.04 AM.png

so we run dbt as an RPC server and dagster triggers production dbt processes by communicating with the server. In practice, our pipelines tend to look something like the following:

most of the time, we are triggering some form of data ingestion and/or transformation to happen pre-dbt, and then triggering a number of dbt processes to happen after (sometimes in parallel, as shown above)

Screen Shot 2020-10-14 at 11.22.10 AM.png

we surface all dbt flags via the Dagster UI so that end users can dynamically change the configuration of run as well:

Screen Shot 2020-10-14 at 11.23.40 AM.png

additionally, we surface all dbt logs through the Dagster UI:

Thanks for the info.  Still trying to wrap my head around it I think.  Previously I was just able to plug in my dbt model and run it as a solid.  Is there some way to do the equivalent in this new setup?

we only really use the RPC solids so I can only speak to that, but we have a single solid `dbt_rpc_run_and_wait` that we alias in a variety of different places to run 1 to n number of models

so the "plugging in" of your dbt model would be equivalent to invoking the `dbt_rpc_run_and_wait` solid with the configuration set to just run your single model

for reference: <https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/rpc/solids.py#L214-L313>

```from dagster_dbt import create_dbt_run_solid

dbt_parse_send_logs = create_dbt_run_solid(
    project_dir='path/to/my_dbt_model/',
    name='dbt_model_name',
)```

This is basically what I was doing before

oh yeah the way I mentioned above is a more dynamic way of doing what you're doing

the model selection is just configuration on the solid itself

So do I have to set up and have dbt running as a separate server now to do the RPC?

yeah, RPC requires dbt to be running as a server

but thats just one way to run dbt via Dagster

interface looks similar though: <https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cli/solids.py#L96-L157>

Ok I'll check these resources out and will circle back with you if I still end up stuck.  Thanks again!

<@UUYAH8SFJ> You are right that `dagster-dbt` has changed quite a bit recently. The <https://docs.dagster.io/_apidocs/libraries/dagster_dbt|API docs> are a bit terse, so we have diff that is inflight that will add a section on `dagster-dbt` in our <https://docs.dagster.io/examples|examples page> with code snippets in GitHub.

as <@ULW9GTUGP> mentioned, there are solids for the both dbt RPC server and the dbt CLI.

in the case of dbt CLI solids, you would pass in CLI options via the solid configs, such as `--project-dir`, `--models`, `--exclude`, etc.