I'm working roughly from the MDS demo in the examp...
# ask-community
I'm working roughly from the MDS demo in the example folder of the git repo. I feel like I should be able to strip out the pandas_io_manager, because DBT should just be executing SQL statements against the database, but if I try to remove it I get:
Copy code
docker-dev-dagster-pipelines     |   File "/usr/local/lib/python3.9/site-packages/dagster/core/asset_defs/asset_group.py", line 755, in _validate_resource_reqs_for_asset_group
docker-dev-dagster-pipelines     |     raise DagsterInvalidDefinitionError(
docker-dev-dagster-pipelines     | dagster.core.errors.DagsterInvalidDefinitionError: Output 'study_list_completion_mvw' with AssetKey 'AssetKey(['raw', 'study_list_completion_mvw'])' requires io manager 'pandas_io_manager' but was not provided on asset group. Provided resources: ['dbt', 'io_manager']
Could anyone explain to me why the SQL side of the DBT integration requires an IO manager?
🤖 1
hi @George Pearse! good question -- this io manager doesn't actually do anything on the dbt side (which is confusing). It's purely used to tell downstream assets that take these dbt models as input how to load them. If you don't have anything downstream of the dbt models, then there's no need for an io manager.
this is definitely an issue that we're thinking about -- it's definitely not the most ergonomic pattern to have to anticipate how downstream assets will want to load these dbt assets.
Fair enough, sure I will use its core functionality soon. My only concern was if I was unnecessarily loading lots of data into python between the DBT models.
gotcha! 👍
it's surprising to me that you're seeing that error @George Pearse. Did you make sure to remove
in the invocation of
Ah think you've got it
One sec
Yeah that's done it. Next PR to the repo will probably be bringing it back in tbh but nice to understand.