How can I tell dagster to not use an IO Manager at all for a specific asset? I'm using Spark & managing i/o myself within the op but Dagster is still logging events, reading nothing, writing nothing, and looking for an empty file as an input dependency every time. I've tried:

@asset(io_manager_def=None)

but that doesn't do it.

My understanding is that the IO manager is an integral part of the execution in Dagster, so most likely you cannot simply exclude it (I’m not 100% on this). But you can create your “noop” io manager, it’s pretty easy to subclass the builtin

IOManager

class (which will be still called, so you’ll see the logs). Also, you can use the

non_argument_deps

keywords to specify logical dependencies on assets without actual data dependency. I believe in this case Dagster won’t run

load_input

on the upstream asset’s io_manager (but

handle_output

will still be called when the upstream asset is materialized). https://docs.dagster.io/concepts/assets/software-defined-assets#non-argument-dependencies

hi @Kevin Schaich! to tell Dagster that an asset will not return any python object that will need to be managed by an IOManager, you'll want the Nothing type. In short, the easiest way to "turn off" the IOManager is by annotating your asset function with a

None

return type:

@asset
def my_asset() -> None:
    pass

but you can also do this by setting the

dagster_type

of the asset to `Nothing`:

@asset(dagster_type=Nothing)
def my_asset():
    pass

@owen Thanks for the clarification! Is the dagster_type=Nothing equivalent with returning Output(None) from the asset function? Or the latter does invoke the io-manager but with a value of None?

no problem — the latter invokes the io manager with a value of None