This channel is a place where you can get the latest updates on using Dagster with dbt, ask questions, and discuss with other members of the community. Looking to get started using Dagster with dbt? • ⚙️ Check out our guide with Dagster and dbt Core • ☁️ Check out our guide with Dagster and dbt Cloud • 📚 Take a look at our API docs for `dagster-dbt`

We’ll be releasing partition support for our dbt Cloud integration tomorrow in 0.17.4! Here’s a sneak peek of what to expect: • When you materialize the assets in the Dagster Cloud UI, you’ll specify what partitions you want to materialize. You can define these partitions when you generate your software defined assets from your dbt Cloud job. • The partition will be mapped to a variable that’s passed into your dbt Cloud job. You can use this variable however you want in your dbt project. This mapping is customizable in code.

Does dagster have native support for dbt tests? For example, would like to • prevent pipeline from running if it fails dbt test DQ check • visually see in the UI if any assets are failing any DQ checks from dbt test

Here’s also another sneak peek of our subsetting support for dbt Cloud: • You can now specify individual assets, managed by a dbt Cloud job, that you want to materialize. When you materialize them, we will pass the relevant filters when triggering the dbt Cloud job to materialize your selected subset.

👋

Hello 👋, dagster-dbt does not work correctly when we change the target-path (which is useful when dbt is not allowed to write in the project dir). I'm on the latest version 0.17.5 The value of

target-path

in the

dbt_project.yml

is not used • a workaround is to set the config target_path for

dbt_cli_resource

, but I think many dbt users will want to use

dbt_project.yml

Is there a future plan to support multiple dbt cmd statements in a job? Referring to

dbt Cloud

resource and as Assets. For e.g a

dbt run-operation

before or after

dbt build

in a job?

Hi guys, I have trouble trying to use define_asset_job to find specific DBT model to run, only "*" and specific single asset "my_first_dbt_model" while i want to to run the whole "example" model. After lots of time spent on reading document, I find writing job using dbt_run_op() with dbt_cli_resource works fine. However, they are in old document, so I am a bit scared of them being depreciated.

Hey all, is there a way with dagster_dbt to have it materialize assets that match a DBT tag? From what I've seen you have to use the pre-defined asset name that's given (with some customization there like prefix). Or if I want to do something like this am I better off building something custom that runs the job from the CLI resource and determines which assets were materialized? The goal is to be able to do arbitrary dbt runs and have Dagster show the corresponding assets as materialized

so I have a working dbt set of models and when I try to use the load_assets_from_dbt_project poointing to my dbt project I get the error Error loading repository location myproject_dagsterdagster. core.errors.DagsterImportError Encountered ImportError: `cannot import name 'load_assets_from_dbt_project' from partially initialized module 'myproject_dagster' (most likely due to a circular import)

not sure what to check

this is using dbt core and dagster locally

Hello I was wandering if there was a way to tell Dagster that DBT incremental models are partitionned assets ?

Hey, we are relying on this PR to be merged/released, as it enables use to load 2 (or more) dbt projects into the same dagster repo. Do you have an indication on when it will be released?

so I was able to get my DBT models into dagster, but how do I also get the seeds into dagster

Hi, are there any examples on how to get custom run metadata from dbt runs? I want to get the row count of the model after its materialization (and plot it in the UI)

Quick query: Does anybody aware about how to execute

dbt_metrics_default_calendar

dbt model from

dbt_packages/metrics

. I am using DBT metrics to create few models. So those metrics are depends upon

dbt_metrics_default_calendar

dbt model, from

dbt_packages/metrics

. But I am unable to execute the

dbt_metrics_default_calendar

from dagster asset. Running dbt command is not picking

dbt_metrics_default_calendar

. Its picking all the custom models defined by me in the

DBT_ROOT_DIR

path except default calendar model from the package.

Warehouse: Snowflake

dbt_metrics: 1.3.0 --version

dbt=1.3.1

Hi...Does the

FreshnessPolicy

&

build_asset_reconciliation_sensor

as outlined apply to dbt Cloud assets? I'm aware that if you select individual assets to materialize, Dagster will override the dbt cloud job with that selection, so I'm wondering Dagster takes the same approach with freshness and reconciliation. E.g. a non-dbt cloud asset, A, is stale and dbt cloud asset, B, is dependent on A. Asset B has dbt children C & D but dbt cloud job also has E, F and G which has no dependencies). Will the reconciliation sensor and FreshnessPolicy work and result in Dagster triggering a dbt build for only B, C and D? https://docs.dagster.io/guides/dagster/scheduling-assets

Hi, everyone! I had a question concerning the materialization of dbt assets and SDAs in the same job I have 2 group of assets, 1 with dbt assets and another one with SDAs that depend on the dbt assets (for instance, I have 2 dbt assets,

dbt_a

and

dbt_b

and 2 SDAs,

sda_a

and

sda_b

with

dbt_a -> sda_a

and

dbt_b -> sda_b

My concern is that when I'm trying to build a job with

define_asset_job("job", selection=AssetSelection.groups("dbt") | AssetSelection.groups("sda"))

, when a dbt materialization fails, none of the sda materialization is launched because a single step is build for all the dbt assets (I'd prefer if

sda_b

was run if

dbt_b

succeeds, even if

dbt_a

fails) Do you know if there is a way to achieve this?

How can I pass (arbitrary variables i.e. lets say a date variable) to DBT run on DBT core orchestrated via dagster?

I have a dbt cli resource which is configured to use the prod target. (the default one in the dbt config would be dev) but this is overwritten in dagsters resource to use PROD. For the materialization of the dbt run command - this works fine. However, dbt test somehow is falling back to the default configuration.

Hello, I was following this tutorial every step and encounter weird error. Every module install i try to pip install on my local also it should be no problem not sure what is going on.

dagster._core.errors.DagsterImportError: Encountered ImportError: `cannot import name 'duckdb_pandas_io_manager' from 'dagster_duckdb_pandas' (C:\Users\Punyawee\AppData\Local\Programs\Python\Python39\lib\site-packages\dagster_duckdb_pandas\__init__.py)` while importing module tutorial_dbt_dagster. Local modules were resolved using the working directory `D:\Playground\testest\my-dagster-project\tutorial_finished`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets.
  File "C:\Users\Punyawee\AppData\Local\Programs\Python\Python39\lib\site-packages\dagster\_grpc\server.py", line 230, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "C:\Users\Punyawee\AppData\Local\Programs\Python\Python39\lib\site-packages\dagster\_grpc\server.py", line 104, in __init__
    loadable_targets = get_loadable_targets(
  File "C:\Users\Punyawee\AppData\Local\Programs\Python\Python39\lib\site-packages\dagster\_grpc\utils.py", line 53, in get_loadable_targets
    else loadable_targets_from_python_package(package_name, working_directory)
  File "C:\Users\Punyawee\AppData\Local\Programs\Python\Python39\lib\site-packages\dagster\_core\workspace\autodiscovery.py", line 49, in loadable_targets_from_python_package
    module = load_python_module(
  File "C:\Users\Punyawee\AppData\Local\Programs\Python\Python39\lib\site-packages\dagster\_core\code_pointer.py", line 141, in load_python_module
    raise DagsterImportError(
The above exception was caused by the following exception:
ImportError: cannot import name 'duckdb_pandas_io_manager' from 'dagster_duckdb_pandas' (C:\Users\Punyawee\AppData\Local\Programs\Python\Python39\lib\site-packages\dagster_duckdb_pandas\__init__.py)
  File "C:\Users\Punyawee\AppData\Local\Programs\Python\Python39\lib\site-packages\dagster\_core\code_pointer.py", line 136, in load_python_module
    return importlib.import_module(module_name)
  File "C:\Users\Punyawee\AppData\Local\Programs\Python\Python39\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "D:\Playground\testest\my-dagster-project\tutorial_finished\tutorial_dbt_dagster\__init__.py", line 1, in <module>
    from .repository import tutorial_dbt_dagster
  File "D:\Playground\testest\my-dagster-project\tutorial_finished\tutorial_dbt_dagster\repository.py", line 4, in <module>
    from dagster_duckdb_pandas import duckdb_pandas_io_manager

https://github.com/dagster-io/dagster/tree/master/examples/tutorial_dbt_dagster

Is anyone using dbt Cloud today and considering moving out to Dagster due to the price increase they announced yesterday? I feel like the value we get from their Cloud is too low for the price. Moving to Dagster will bring some benefits but I’m not sure Dagster has the same easy access to artifacts, build errors and observability features.

Hey team, I found this in my Dagster cloud deployment this morning. I moved the the new

Definitions

api and I got this. I don't think it is Dagster since it's dbt's api limits. Any ideas how I can get around this? Or any others seeing the same thing?

Hi everyone, very new to Dagster and trying to conceptualize how I might setup the v0 dagster <> dbt integration. All the examples I can find have the project in the same repo as the Dagster code. Is it possible to have the dbt project in its own repository? I was imagining having a dbt repo that builds + pushes image to aws ECR, then a dagster job that fetches latest image and runs

dbt run

you can fine-tune the asset key using the parameters of the load assets from dbt method of dagster

Hey - how does dagster handle the output of dbt

view

tables? I noticed after running such a table that it produces a log like:

op 'test_view_7d9e8' did not fire outputs {'test_view'}

And no assets are actually materialized. Is this expected behavior?

Hi friends, what's the

io_manager

that's used when we

load_assets_from_dbt_project

? I want to be able to use my dbt assets like this

@asset
def pull_table_as_dataframe(context, my_dbt_table_one):
     ab = my_dbt_table_one.rename(columns={...})
     ## Now use the ab object for other things
     return whatever

Where

my_dbt_table_one

was loaded using

load_assets_from_dbt_project

. I know I need to configure an

io_manager

for this. Wondering if anyone has used a custom IO manager when loading assets from dbt?

Has anyone ever tried enhancing the manifest parsing function to grab more config properties for assets, such as tags, materialized, and meta? Seems like it could be an easy win to improve dagsters utility as a data catalog, but figured I'd check here in case it's been tried and there's an obvious pitfall

hi guys. I want to ask, how to do dbt cli command like "$ dbt run --full-refresh --select my_incremental_model+" in dagster as a job so we can run it to full refresh the "my_incremental_model" ???