Hello everyone, I have a small issue with the IO_m...
# ask-community
Hello everyone, I have a small issue with the IO_manager used by my assets, basically I create an asset mentioning the IO_manager_key and the required resource key for this IO_manager, but when I run the materialization, it just uses the default key "io_manager" and takes the default FilesystemIOManager. Asset definition :
def build_output_asset (asset_name: str, key_prefix: list, ins: dict) -> asset:
name = asset_name,
key_prefix = key_prefix,
required_resource_keys = {"custom_io_manager"},
io_manager_key = "custom_io_manager",
group_name = "outputs",
ins = {"upstream": AssetIn(**ins)}
def asset_def(context, upstream) -> None:
return asset_def
Resources definition:
resources_definitions = {
"dbt": dbt_cli_resource.configured({
"project_dir": DBT_PROJECT_PATH,
"profiles_dir": DBT_PROFILES
"custom_io_manager": bigquery_pandas_io_manager.configured({
"project": GCP_PROJECT
"io_manager": FilesystemIOManager(),
Error when I run materialization : dagster._core.errors.DagsterExecutionLoadInputError: Error occurred while loading input "upstream" of step "dbt_mf__weather_gcs":: FileNotFoundError: [Errno 2] No such file or directory: '/path/to/a/directory/storage/another_path/path/path' I confirm that when I replace the custom_io_manager as the default IO manager my pipeline works but it messes my code a little bit . Troubleshot a bit further, basically the loaded input is using the key "io_manager" and the output is using the "custom_io_manager", isn't there a way to force using the same "custom_io_manager" in both operations ? Thanks in advance
When loading the input, it will use the IO manager that is set on the upstream asset. Are you setting "custom_io_manager" on the upstream asset? Would this make sense to do?
🌈 1
Hi @sandy, the upstream asset is a dbt asset so I assume it just uses the default io manager. This can be a workaround yes since dbt runs are independent from the io_manager
I just wrote up a long-form answer on this: https://github.com/dagster-io/dagster/discussions/14648 I think the second option might be relevant to what you're looking for
🌈 1