Hi All, I want to integrate dagster-dbt and I use ...
# ask-community
f
Hi All, I want to integrate dagster-dbt and I use
load_assets_from_dbt_project()
, but Asset is not found. it is not showing up in the console where in my local I can see the lineage. dagster version = 0.15.8 deployment in ubuntu compute engine VM GCP
Copy code
dagster_repo
+ dbt_repository.py
dbt
+ config
  + dbt_project.yml
  + profile.yml
+ model
+ macros
in dbt_repository.py, the code looks like this 👇
Copy code
DBT_PROJECT_DIR = '/{ROOT_DIR}/dbt/config'
DBT_PROFILES_DIR = '/{ROOT_DIR}/dbt/config'

dbt_assets = load_assets_from_dbt_project(
    DBT_PROJECT_DIR,
    profiles_dir=DBT_PROFILES_DIR,
    use_build_command=True,
    select="'*'"
)
y
hey Fanny, where exactly did you mean by the “console”?
f
in the UI I mean.
y
what’s in your
repository
definition? did you include
dbt_assets
in it or did you use things like
load_assets_from_package_module
to load the entire module/package?
f
I only use
load_asset_from_dbt_project
and,
Copy code
data_a = define_asset_job("data_a", 
                    selection=AssetSelection.keys(
                        AssetKey(["data_a", "data_a"])))
call the asset like this ☝️
then
Copy code
@repository
def transformation_layer():
    return with_resources(
        dbt_assets,
        resource_defs={
            "dbt": dbt_cli_resource.configured(
                {"project_dir": DBT_PROJECT_DIR, "profiles_dir": DBT_PROFILES_DIR}
            ),
        },
    )
y
in your dbt assets:
Copy code
dbt_assets = load_assets_from_dbt_project(
    DBT_PROJECT_DIR,
    profiles_dir=DBT_PROFILES_DIR,
    use_build_command=True,
    select="'*'"
)
you had
Copy code
select="'*'"
was it intentional to select
'*'
as opposed to
*
?
f
I remember copy it from somewhere. but previously I tried
Copy code
dbt_assets = load_assets_from_dbt_project(
    DBT_PROJECT_DIR,
    profiles_dir=DBT_PROFILES_DIR,
)
also not working.
y
Copy code
DBT_PROJECT_DIR = '/{ROOT_DIR}/dbt/config'
should be
Copy code
DBT_PROJECT_DIR = '/{ROOT_DIR}/dbt'
f
now I got this error
Copy code
dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2)
File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 227, in __init__
    self._loaded_repositories = LoadedRepositories(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 101, in __init__
    loadable_targets = get_loadable_targets(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_grpc/utils.py", line 33, in get_loadable_targets
    else loadable_targets_from_python_file(python_file, working_directory)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/workspace/autodiscovery.py", line 26, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file, working_directory)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/code_pointer.py", line 86, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_seven/__init__.py", line 51, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/dagster/repo_sync/data_engineering/dbt_repository.py", line 20, in <module>
    dbt_assets = load_assets_from_dbt_project(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 424, in load_assets_from_dbt_project
    manifest_json, cli_output = _load_manifest_for_project(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/asset_defs.py", line 41, in _load_manifest_for_project
    cli_output = execute_cli(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_dbt/cli/utils.py", line 139, in execute_cli
    raise DagsterDbtCliFatalRuntimeError(
y
the error usually means the path to the dbt project isn’t correctly set. here’s how we load the dbt project into assets: https://github.com/dagster-io/dagster/blob/master/examples/assets_dbt_python/assets_dbt_python/repository.py#L17 where the folder structure is similar to yours.
f
just tried to follow the code
Copy code
DBT_PROJECT_DIR = file_relative_path(__file__, "../dbt/config")
DBT_PROFILES_DIR = file_relative_path(__file__, "../dbt/config")
but still did not work (no Asset loaded to the repository). do u think its because of version? I use 0.15.8
y
could you try with:
Copy code
DBT_PROJECT_DIR = file_relative_path(__file__, "../dbt") # this should point to the dbt project folder, not the config folder
DBT_PROFILES_DIR = file_relative_path(__file__, "../dbt/config")
f
got this error again.
Copy code
dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2)
y
cc @owen does the error ring any bell?
o
hi @Fanny Wiryana! sorry you're running into this. In a more recent update to the dagster-dbt library, we made it so that a more descriptive error message would be shown when you encountered this error, which would definitely help tracking down the exact cause here. There's all sorts of potential issues that might cause this (for example, if the connector library like dbt-postgres or dbt-postgres is not installed on the machine that's running the code). If it's easy for you to do, then upgrading to the newest version of dagster-dbt might be a good option. regardless of your dagster-dbt version, this more descriptive error message also gets displayed in the terminal that dagit is running in. it sounds like you're running your dagster deployment on a remote machine, so it might not be easy to access these logs, but if you're able to, this would also be a good thing to check. You can also try running dagit on your local machine (
dagit -f path/to/dbt_repository.py
), to view the logs there
👍 1