FĂ©lix Tremblay
02/16/2023, 9:31 PMload_assets_from_dbt_project
, fails if dbt deps
are not installed.
When using Serverless with FAST_DEPLOYS, this is a big issue, since there's no way to install the deps using dagster_cloud_post_install.sh
.
Can you please add the argument run_deps_command: bool
(or ``install_deps: bool``, or similar) to load_assets_from_dbt_project
; when set to True, it would simply run a command like run_command(["dbt", "deps", "--project-dir", DBT_PROJECT_DIR])
before running dbt run
/ dbt build
Adam Bloom
02/17/2023, 3:11 PMdbt
command should be available. You really don’t want to run dbt deps
every time before running dbt run/build - it will slow down execution and insert potential for runs of the same dagster build to be less deterministic.FĂ©lix Tremblay
02/17/2023, 6:47 PMdagster_cloud_post_install.sh
is not available for fast-deploys (hooks only work for slow-deploys)FĂ©lix Tremblay
02/17/2023, 7:36 PMFĂ©lix Tremblay
02/17/2023, 7:41 PMFĂ©lix Tremblay
02/20/2023, 7:07 PMShalabh Chaturvedi
02/21/2023, 5:40 PMdeploy.yml
to run dbt deps
after the fast deploys checkout:
- name: Checkout
if: env.ENABLE_FAST_DEPLOYS == 'true'
uses: actions/checkout@v3
with:
ref: ${{ github.head_ref }}
path: project-repo
# Add these two lines:
- run: "pip install dbt-core"
- run: "dbt deps --project-dir $GITHUB_WORKSPACE/project-repo/dbt_project"
FĂ©lix Tremblay
03/01/2023, 6:36 PMFĂ©lix Tremblay
03/01/2023, 6:38 PMFĂ©lix Tremblay
03/01/2023, 6:45 PMShalabh Chaturvedi
03/02/2023, 10:40 PMDo you think it would make sense to put the 'run: "dbt deps ..." ' step after the python project installation step? (This way dbt would already be installed and would have the version that is pinned in setup.py)Hi FĂ©lix, that's a good point regarding the version. When using fast deploys, the installation of dbt-core and other dependencies doesn't really happen until the code is deployed in our cloud. I will discuss the best way to do this with the team. I will also bring your suggestions about better docs (agree!) and automatic
deps
handling to the team discussion.FĂ©lix Tremblay
03/03/2023, 12:58 PM