https://dagster.io/ logo
Join Slack
Powered by
Hi, I'm having some issues adding a DBT project in...
# ask-community
g
Hi, I'm having some issues adding a DBT project into my Dagster serverless cloud deployment. The DBT and Dagster projects are in separate repositories. I've successfully added the project to my CI and install all 
dbt deps
, However when it comes to actually deploying to dagster cloud it seems the runtime cannot find the DBT project.
Error message I receive from the 
Python Executable Deploy
step:
Copy code
Error loading data-orchestrator: {'__typename': 'PythonError', 'message': "dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2): Usage: dbt ls [OPTIONS] Try 'dbt ls -h' for help.  Error: Invalid value for '--project-dir': Path '/venvs/5186e7e1f131/lib/python3.8/site-packages/working_directory/root/data_orchestrator/assets/dbt/../../data-transforms' does not exist.\n"
CI config I'm currently trying to run on GitHub
Copy code
name: Serverless Branch Deployments
on:
  pull_request:
    types: [opened, synchronize, reopened, closed]
    
concurrency:
  # Cancel in-progress deploys to same branch
  group: ${{ github.ref }}/branch_deployments
  cancel-in-progress: true
env:
  DAGSTER_CLOUD_URL: "<http://milkmoovement.dagster.cloud>"
  DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
  ENABLE_FAST_DEPLOYS: 'true'
  PYTHON_VERSION: '3.8'
  DAGSTER_CLOUD_FILE: 'dagster_cloud.yaml'

jobs:
  dagster_cloud_default_deploy:
    name: Dagster Serverless Deploy
    runs-on: ubuntu-20.04
    outputs:
      build_info: ${{ steps.parse-workspace.outputs.build_info }}
    
    steps:
      - name: Prerun Checks
        id: prerun
        uses: dagster-io/dagster-cloud-action/actions/utils/prerun@v0.1

      - name: Launch Docker Deploy
        if: steps.prerun.outputs.result == 'docker-deploy'
        id: parse-workspace
        uses: dagster-io/dagster-cloud-action/actions/utils/parse_workspace@v0.1
        with:
          dagster_cloud_file: $DAGSTER_CLOUD_FILE

      - name: Checkout for Python Executable Deploy
        if: steps.prerun.outputs.result == 'pex-deploy'
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}
          path: project-repo
          submodules: 'true'
          token: ${{ secrets.CROSS_REPO_GH_TOKEN }}

      # Install dbt deps
      # <https://stackoverflow.com/questions/75568254/importerror-cannot-import-name-parametersource-from-click-core>
      - run: pip install --no-deps --ignore-installed click
      - run: pip install dbt-core
      - run: dbt deps --project-dir $GITHUB_WORKSPACE/project-repo/data-transforms
          
      - name: Python Executable Deploy
        if: steps.prerun.outputs.result == 'pex-deploy'
        uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@v0.1
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/$DAGSTER_CLOUD_FILE"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "${{ env.PYTHON_VERSION }}"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
The 
data-transforms
DBT project repo is added as a submodule in the 
data-orchestrator
Dagster repository and updated every time a change is made in 
main
in the DBT project
I don't seem to have any issues with the initial asset test locally, everything seems to go wrong at the python executable deploy step
Full error
Copy code
Error loading data-orchestrator: {'__typename': 'PythonError', 'message': "dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2): Usage: dbt ls [OPTIONS] Try 'dbt ls -h' for help.  Error: Invalid value for '--project-dir': Path '/venvs/5186e7e1f131/lib/python3.8/site-packages/working_directory/root/data_orchestrator/assets/dbt/../../data-transforms' does not exist.\n", 'stack': ['  File "/venvs/fe545ff9ddf9/lib/python3.8/site-packages/dagster/_grpc/server.py", line 278, in __init__\n    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(\n', '  File "/venvs/fe545ff9ddf9/lib/python3.8/site-packages/dagster/_grpc/server.py", line 122, in __init__\n    loadable_targets = get_loadable_targets(\n', '  File "/venvs/fe545ff9ddf9/lib/python3.8/site-packages/dagster/_grpc/utils.py", line 57, in get_loadable_targets\n    else loadable_targets_from_python_package(package_name, working_directory)\n', '  File "/venvs/fe545ff9ddf9/lib/python3.8/site-packages/dagster/_core/workspace/autodiscovery.py", line 48, in loadable_targets_from_python_package\n    module = load_python_module(\n', '  File "/venvs/fe545ff9ddf9/lib/python3.8/site-packages/dagster/_core/code_pointer.py", line 135, in load_python_module\n    return importlib.import_module(module_name)\n', '  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n', '  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import\n', '  File "<frozen importlib._bootstrap>", line 991, in _find_and_load\n', '  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked\n', '  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked\n', '  File "<frozen importlib._bootstrap_external>", line 843, in exec_module\n', '  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed\n', '  File "/venvs/5186e7e1f131/lib/python3.8/site-packages/working_directory/root/data_orchestrator/__init__.py", line 11, in <module>\n    from data_orchestrator.assets.dbt import DBT_PROFILES, DBT_PROJECT_PATH\n', '  File "/venvs/5186e7e1f131/lib/python3.8/site-packages/working_directory/root/data_orchestrator/assets/dbt/__init__.py", line 8, in <module>\n    dbt_assets = load_assets_from_dbt_project(\n', '  File "/venvs/fe545ff9ddf9/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 520, in load_assets_from_dbt_project\n    manifest_json, cli_output = _load_manifest_for_project(\n', '  File "/venvs/fe545ff9ddf9/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 76, in _load_manifest_for_project\n    cli_output = execute_cli(\n', '  File "/venvs/fe545ff9ddf9/lib/python3.8/site-packages/dagster_dbt/cli/utils.py", line 229, in execute_cli\n    for event in _core_execute_cli(\n', '  File "/venvs/fe545ff9ddf9/lib/python3.8/site-packages/dagster_dbt/cli/utils.py", line 151, in _core_execute_cli\n    raise DagsterDbtCliFatalRuntimeError(messages=messages)\n']}
I think I got around this by adding
Copy code
package_data={
        "data_orchestrator": ["./" + path for path in glob.glob("./data-transforms/**", recursive=True)],
    },
into my 
setup
call in the 
setup.py
file. Still testing
j
https://github.com/dagster-io/dagster/discussions/12180 Hi, I'm not sure but I think it could help.
When I deploy to cloud serverless, I disabled the 
ENABLE_FAST_DEPLOYS
and then create the 
dagster_cloud_post_install.sh
with execute permission
Copy code
# dagster_cloud_post_install.sh

# install dbt deps
echo "installing dbt deps"
cd /opt/dagster/app/dbt && dbt deps
g
The problem was the actual DBT folder was not being added into the image, but I think the 
package_data
setting fixed that. Resolving other issues before I can confirm.
👍 1
Can't seem to get past this error when loading dbt assets
Copy code
dagster._core.errors.DagsterImportError: Encountered ImportError: `No module named 'data_orchestrator.assets.dbt.adapters'` while importing module data_orchestrator. Local modules were resolved using the working directory `/venvs/a990fd245e8d/lib/python3.8/site-packages/working_directory/root`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets.
Don't see any references to this in the code anywhere 
data_orchestrator.assets.dbt.adapters
So, incase anyone else was reading this thread it was because I was trying to import assets created by 
load_assets_from_dbt_project
with a 
load_assets_from_package_module
call in the root folder for my project... just use the output variable from calling 
load_assets_from_dbt_project
directly when adding it to 
Definitions
under the 
assets
input.
2 Views
Open in Slack
PreviousNext
Powered by