Hey I'm working with the new `@dbt_assets` syntax ...
# integration-dbt
Hey I'm working with the new
syntax and I have to provide path to manifest. Before migration to new syntax I was using
and manifest.json was in gitignore and not commited to repo. With the new syntax I have to commit the manifest and after few days of using it like this it is not ideal and manifest always causes merge conflicts. Is there any plan to add similar functionality like
to new
decorator ? I'm thinking of reimplementing
and running it before
. I only have like 70 models. Is there any obvious disadvantage to this approach?
plus1 1
following. but another way is to save the
file in a different location (S3 etc.)
Good tip, I like the simplicity of
It runs
dbt ls
on each deploy and creates manifest for it self without dependency to any external service.
dagster-dbt project scaffold
, we emulate the behavior of
by creating the manifest at runtime. You can see the implementation here. • We use the presence of an env var
to determine whether to build a manifest. • This allows for workflows in development where you run
locally so that you can work while simultaneously updating your dbt and dagster code. And in production, you can produce the manifest in your build system, when you deploy your code. You don’t need to commit it to your repository. • We recommend using a precompiled manifest because it is more efficient. Otherwise, everytime the code location loads, or whenever a Dagster job is kicked off to materialize your assets, you have to compile your manifest. Depending on the scale of your project, this could be an expensive operation. • By compiling it in the build system, you just do this compilation operation once for your deployed code.
We're not yet using the new DBT API, but we've been having much better performances on job loading times and less loading errors by using
compared to
. • Our DBT project is an independent Git project. Its
directory, that receives the manifest, is git-ignored. • It is included in our Dagster git project as a git submodule. • If DBT models are updated, we create a new release of the DBT project and the Dagster project with its updated submodule. • Our Dagster project deployment/installation script runs
dbt ls
to generate the manifest in the
folder of the submodule, which remains ignored. The only difficulty was to find the correct git commands to update the git submodules
git submodule sync && git submodule update --init --force --recursive
, but since this finding (based on heavy archeological and experimental work) it has been working perfectly.
👀 2