https://dagster.io/ logo
Title
g

Gabriel Simmons

05/26/2021, 6:32 AM
Hi all! Is it possible to specify an
IOManager
for the output of a composite solid? A toy version of my scenario below. I would like to be able to use
func_b
as part of several composite solids, and I don’t know that I always want to use
my_custom_io_manager
with
func_b
. I do know that I always want to use
my_custom_io_manager
with the composite solid
func_b_a
. With the setup below, running
func_b_a_pipeline
does not use
my_custom_io_manager
and uses the default in-memory manager instead. When I move the
OutputDefinition
to the decorator for
func_b
, the custom io manager is used, but this means that I’m always using this io manager whenever I use func_b, which is not desirable for me. I’m new to Dagster, any ideas on workarounds are appreciated.
@solid
def func_a() -> ReturnsSomething:
    ...

@solid
def func_b(input) -> AlsoReturnsSomething:
    ...

@composite_solid(
    output_defs=[OutputDefinition(io_manager_key="key_for_my_custom_io_manager")]
)
def func_b_a():
    return func_b(func_a())

@pipeline
def func_b_a_pipeline():
    func_b_a():
c

chris

05/26/2021, 2:07 PM
The outputs of composite solids are "mapped" from the underlying solids that produce them, meaning that the io manager specified on the underlying solid is the one that will always be used to persist that output.
Currently, providing an IO manager key on output definition of composite solid is ignored, which is not a great state. I just put up an issue to track this: https://github.com/dagster-io/dagster/issues/4214
❤️ 1
As far as potential workarounds, I think separate modes for each of the possible IO managers that you want to use with
func_b
could fit this use case; ie one mode where "key_for_custom_io_manager" is mapped to
mem_io_manager
, and another where it is mapped to
custom_io_manager
.
👍 1
g

Gabriel Simmons

05/27/2021, 4:22 AM
@chris Thank you, a user warning would have definitely been helpful!