https://dagster.io/ logo
Join Slack
Powered by
hi guys, I tried to install dagster-dbt and deploy...
# ask-community
f
hi guys, I tried to install dagster-dbt and deploy one repository of it but keep getting error
Copy code
FileNotFoundError: [Errno 2] No such file or directory: 'dbt'
dagster version is 0.15.8
dagster bot responded by community 1
this is the full error message
Copy code
FileNotFoundError: [Errno 2] No such file or directory: 'dbt'
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 227, in __init__
    self._loaded_repositories = LoadedRepositories(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 101, in __init__
    loadable_targets = get_loadable_targets(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/utils.py", line 33, in get_loadable_targets
    else loadable_targets_from_python_file(python_file, working_directory)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/workspace/autodiscovery.py", line 26, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file, working_directory)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/code_pointer.py", line 86, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_seven/__init__.py", line 51, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/dagster/repo_sync/data_engineering/dbt_repository.py", line 20, in <module>
    dbt_assets = load_assets_from_dbt_project(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 424, in load_assets_from_dbt_project
    manifest_json, cli_output = _load_manifest_for_project(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 41, in _load_manifest_for_project
    cli_output = execute_cli(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/cli/utils.py", line 102, in execute_cli
    process = subprocess.Popen(
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
this is version installed
Copy code
dagit==0.15.8
dagster==0.15.8
dagster-airbyte==0.15.8
dagster-cron==0.11.16
dagster-dbt==0.15.8
dagster-graphql==0.15.8
dagster-pandas==0.15.8
dagster-postgres==0.15.8
dagster-slack==0.15.8
db-dtypes==1.0.2
dbt-bigquery==1.2.0
dbt-core==1.2.0
dbt-extractor==0.4.1
dbt-postgres==1.2.0
a
This sounds like a path issue with your dbt resource. looks like you're using the dbt CLI resource, so somewhere you should have a configuration for it...perhaps something like this:
Copy code
dbt_configured = dbt_cli_resource.configured(
    {
        "project_dir": DBT_PROJECT_DIR,
        "profiles_dir": DBT_PROFILES_DIR,
    }
)
I suspect that whatever path you have configured here happens to work as expected locally, but does not work in your deployed user code
the path may just be "dbt" too, given the error
f
yes it worked in my local. my repo look like
Copy code
data-eng
__ data_engineering
____ dbt_repository
__ dbt_transform
____ dbt.yaml
and this the code
Copy code
ROOT_DIR = "/opt/dagster/repo_sync"
DBT_PROJECT_DIR = f"{ROOT_DIR}/dbt_transform"
DBT_PROFILES_DIR = f"{ROOT_DIR}/dbt_transform/config"

dbt_assets = load_assets_from_dbt_project(
    DBT_PROJECT_DIR,
    DBT_PROFILES_DIR,
)
Copy code
@repository
def transformation_layer():
    return with_resources(
        dbt_assets,
        resource_defs={
            "dbt": dbt_cli_resource.configured(
                {"project_dir": DBT_PROJECT_DIR, "profiles_dir": DBT_PROFILES_DIR}
            ),
        },
    )
a
The specifics on how to do this depend on how you're packaging your deployment, but i would suspect this is related
f
what is your suggestion on how to do the deployment? we’ve tried several things on how to define the path but still doesnt work
a
how are you currently doing this packaging? I assume a docker container? if so, what's your entrypoint and how are the files added? You'll probably want to use either something like https://docs.dagster.io/_apidocs/utilities#dagster.file_relative_path or just 
os.getcwd()
I re-read your error - I think I might have misled you. Your error is not about accessing the dbt project/profile - you aren't even getting that far. It's not able to access the dbt executable. Are you perhaps using a virtualenv or otherwise not adding dbt to your PATH?
f
sorry for the late reply over the weekend. yes we are using compute engine in GCP platform, we are using ubuntu. how do I check this part 
not adding dbt to your PATH
in server?
after trying to run 
sudo pip install dbt-bigquery=1.2.0
, now I got new error when loading the repo
Copy code
dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 227, in __init__
    self._loaded_repositories = LoadedRepositories(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 101, in __init__
    loadable_targets = get_loadable_targets(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/utils.py", line 33, in get_loadable_targets
    else loadable_targets_from_python_file(python_file, working_directory)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/workspace/autodiscovery.py", line 26, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file, working_directory)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/code_pointer.py", line 86, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_seven/__init__.py", line 51, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/dagster/repo_sync/data_engineering/dbt_repository.py", line 22, in <module>
    dbt_assets = load_assets_from_dbt_project(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 424, in load_assets_from_dbt_project
    manifest_json, cli_output = _load_manifest_for_project(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 41, in _load_manifest_for_project
    cli_output = execute_cli(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/cli/utils.py", line 139, in execute_cli
    raise DagsterDbtCliFatalRuntimeError(
y
Hi, we also have the same problem. I dunno about the deployment process since it's fully managed by DevOps using GCP Cloud Build. I think that due to permission access but I am not sure since I can't view full log error (below 
raise DagsterDbtCliFataRuntimeError
). I hope anyone could give some clues about this. (We have not used neither environment or container option, it is global setup, place the repository into certain folder in the VM for CI/CD sync)
f
hai @Adam Bloom
2 Views
Open in Slack
PreviousNext
Powered by