https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
f

Félix Tremblay

02/16/2023, 9:31 PM
Hello 👋 ,
load_assets_from_dbt_project
, fails if
dbt deps
are not installed. When using Serverless with FAST_DEPLOYS, this is a big issue, since there's no way to install the deps using
dagster_cloud_post_install.sh
. Can you please add the argument
run_deps_command: bool
(or ``install_deps: bool``, or similar) to
load_assets_from_dbt_project
; when set to True, it would simply run a command like
run_command(["dbt", "deps", "--project-dir", DBT_PROJECT_DIR])
before running
dbt run
/
dbt build
a

Adam Bloom

02/17/2023, 3:11 PM
(Disclaimer: not a dagster cloud user) why wouldn’t post install hooks work? According to docs, these are run after python deps (I.e. dbt) are installed, so the
dbt
command should be available. You really don’t want to run
dbt deps
every time before running dbt run/build - it will slow down execution and insert potential for runs of the same dagster build to be less deterministic.
f

Félix Tremblay

02/17/2023, 6:47 PM
It is because
dagster_cloud_post_install.sh
is not available for fast-deploys (hooks only work for slow-deploys)
Since dbt deps takes less than 2 seconds to re-run, it would not slow down execution that much (and is definitely an improvement compared to not being able to run dbt)
As far as being deterministic, assuming that the dbt packages versions are pinned in the packages.yml, I would not expect a high level of risk
Hi @rex, I hope you're doing well. I was wondering if you might have any thoughts or if you could take a quick look when you have a moment? Thank you!
s

Shalabh Chaturvedi

02/21/2023, 5:40 PM
Hi Félix, for fast deploys all local files from the GitHub workflow are deployed to cloud. Can you please try the following: modify
deploy.yml
to run
dbt deps
after the fast deploys checkout:
Copy code
- name: Checkout
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}
          path: project-repo
          
      # Add these two lines:

      - run: "pip install dbt-core"
      - run: "dbt deps --project-dir $GITHUB_WORKSPACE/project-repo/dbt_project"
f

Félix Tremblay

03/01/2023, 6:36 PM
Hello @Shalabh Chaturvedi, thank you very much for the support! I tested and it works
Do you think it would make sense to put the 'run: "dbt deps ..." ' step after the python project installation step? (This way dbt would already be installed and would have the version that is pinned in setup.py)
I think it would be really helpful for other dbt users if the documentation could include more information and examples on how to deal with dbt deps. Is this something you might consider adding in the future? And is this something that could potentially be handled automatically by the dagster-dbt integration?
s

Shalabh Chaturvedi

03/02/2023, 10:40 PM
Do you think it would make sense to put the 'run: "dbt deps ..." ' step after the python project installation step? (This way dbt would already be installed and would have the version that is pinned in setup.py)
Hi Félix, that's a good point regarding the version. When using fast deploys, the installation of dbt-core and other dependencies doesn't really happen until the code is deployed in our cloud. I will discuss the best way to do this with the team. I will also bring your suggestions about better docs (agree!) and automatic
deps
handling to the team discussion.
f

Félix Tremblay

03/03/2023, 12:58 PM
Thank you @Shalabh Chaturvedi
2 Views