https://dagster.io/ logo
#ask-community
Title
# ask-community
k

Kevin Schaich

02/13/2023, 3:06 AM
How can I tell dagster to not use an IO Manager at all for a specific asset? I'm using Spark & managing i/o myself within the op but Dagster is still logging events, reading nothing, writing nothing, and looking for an empty file as an input dependency every time. I've tried:
Copy code
@asset(io_manager_def=None)
but that doesn't do it.
a

Andras Somi

02/13/2023, 9:07 AM
My understanding is that the IO manager is an integral part of the execution in Dagster, so most likely you cannot simply exclude it (I’m not 100% on this). But you can create your “noop” io manager, it’s pretty easy to subclass the builtin
IOManager
class (which will be still called, so you’ll see the logs). Also, you can use the
non_argument_deps
keywords to specify logical dependencies on assets without actual data dependency. I believe in this case Dagster won’t run
load_input
on the upstream asset’s io_manager (but
handle_output
will still be called when the upstream asset is materialized). https://docs.dagster.io/concepts/assets/software-defined-assets#non-argument-dependencies
o

owen

02/13/2023, 5:55 PM
hi @Kevin Schaich! to tell Dagster that an asset will not return any python object that will need to be managed by an IOManager, you'll want the Nothing type. In short, the easiest way to "turn off" the IOManager is by annotating your asset function with a
None
return type:
Copy code
@asset
def my_asset() -> None:
    pass
but you can also do this by setting the
dagster_type
of the asset to `Nothing`:
Copy code
@asset(dagster_type=Nothing)
def my_asset():
    pass
and @Andras Somi is correct for the other end of the equation -- passing in my_asset to a downstream asset can be done the traditional way of making
my_asset
an argument to your downstream function (and no input loading behavior will be invoked, as dagster knows it is of type "Nothing"), but if you prefer, you can not have it as part of the function signature by using non_argument_deps
a

Andras Somi

02/13/2023, 8:40 PM
@owen Thanks for the clarification! Is the dagster_type=Nothing equivalent with returning Output(None) from the asset function? Or the latter does invoke the io-manager but with a value of None?
o

owen

02/13/2023, 8:55 PM
no problem — the latter invokes the io manager with a value of None
🙌 1
10 Views