Gabriel Simmons
05/26/2021, 6:32 AMIOManager
for the output of a composite solid? A toy version of my scenario below. I would like to be able to use func_b
as part of several composite solids, and I don’t know that I always want to use my_custom_io_manager
with func_b
. I do know that I always want to use my_custom_io_manager
with the composite solid func_b_a
. With the setup below, running func_b_a_pipeline
does not use my_custom_io_manager
and uses the default in-memory manager instead. When I move the OutputDefinition
to the decorator for func_b
, the custom io manager is used, but this means that I’m always using this io manager whenever I use func_b, which is not desirable for me. I’m new to Dagster, any ideas on workarounds are appreciated.
@solid
def func_a() -> ReturnsSomething:
...
@solid
def func_b(input) -> AlsoReturnsSomething:
...
@composite_solid(
output_defs=[OutputDefinition(io_manager_key="key_for_my_custom_io_manager")]
)
def func_b_a():
return func_b(func_a())
@pipeline
def func_b_a_pipeline():
func_b_a():
chris
05/26/2021, 2:07 PMfunc_b
could fit this use case; ie one mode where "key_for_custom_io_manager" is mapped to mem_io_manager
, and another where it is mapped to custom_io_manager
.Gabriel Simmons
05/27/2021, 4:22 AM