Hello everyone, I have case which dagster-dbt run ...
# ask-community
y
Hello everyone, I have case which dagster-dbt run smoothly in localhost, but failed in production, (We save our code in Bitbucket and execute CI/CD using GCP Cloud Build) 1. Module dbt not found We have found interesting threads in Stackoverflow. (https://stackoverflow.com/questions/14295680/unable-to-import-a-module-that-is-definitely-installed) After we implement it (install in server using
sudo pip3 install dbt-bigquery=1.2.0
), it's gone but we got No.2 Error 2. DbtRuntimeFatalError(return code 2) We have tried to simulate this in local computer and mostly it means that the dbt project is failed to load the project due to undefined path or access. I dunno if we can just view full error log since it's truncated in the repository load console/UI (It would be great if we know the full log and get our analysis based on it). So far, we suspect it was permission access which dagster can't load dbt project which located in the same repo folder.
Copy code
REPOSITORY_FOLDER (parent_folder)
1. dagster (child_1)
2. dbt (child_1)
This is the latest log,
Copy code
dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 227, in __init__
    self._loaded_repositories = LoadedRepositories(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 101, in __init__
    loadable_targets = get_loadable_targets(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/utils.py", line 33, in get_loadable_targets
    else loadable_targets_from_python_file(python_file, working_directory)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/workspace/autodiscovery.py", line 26, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file, working_directory)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/code_pointer.py", line 86, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_seven/__init__.py", line 51, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/dagster/repo_sync/data_engineering/dbt_repository.py", line 22, in <module>
    dbt_assets = load_assets_from_dbt_project(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 424, in load_assets_from_dbt_project
    manifest_json, cli_output = _load_manifest_for_project(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 41, in _load_manifest_for_project
    cli_output = execute_cli(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/cli/utils.py", line 139, in execute_cli
    raise DagsterDbtCliFatalRuntimeError(
Hope there is someone who know the workaround. Thanks for you reading time.
y
1. seems like a missing dependency in your CI/CD or prod env 2. did your prod or CI/CD env know of the dbt folder? in our fully featured example (code), we organize dagster and dbt in the same structure and it works properly in our remote deployments.