Sergio Pintaldi
04/04/2023, 11:38 PMmulti_asset
: I want to define a multi asset because the output is generated using the same op/logic. So I went on and define a multi_asset
as below
from dagster import AssetIn, AssetKey, AssetOut, Output, multi_asset
@multi_asset(
name="my_op",
ins={
"in_a": AssetIn("complex_name_for_input_a"),
"in_b": AssetIn("complex_name_for_input_b"),
},
outs={"a": AssetOut(is_required=False), "b": AssetOut(is_required=False)},
internal_asset_deps={
"a": {AssetKey("complex_name_for_input_a")},
"b": {AssetKey("complex_name_for_input_b")},
},
can_subset=True,
)
def my_complex_assets(context, in_a, in_b):
out_value = None
out_name = None
if "a" in context.selected_output_names:
out_name = "a"
out_value = in_a + 1
if "b" in context.selected_output_names:
out_name = "b"
out_value = in_b + 1
if out_value is not None and out_name is not None:
yield Output(value=a + 1, output_name="a")
The issue that I am facing is that when selecting and materialising one output, let's say "a"
, the op needs to read both the inputs, but if they do not exists, it fails. I hoped it needed to read only the one required by the respective output.
With that I was thinking of achieving the logic "if this input do this output". What can I do to achieve this logic? Separate into single assets wrapping the internal logic into a python function?sandy
04/05/2023, 12:10 AMSeparate into single assets wrapping the internal logic into a python function?This seems like a reasonable approach to me Feel free to file a github issue with this request. I think the main question is what Dagster would pass as a parameter value into the function for the inputs that aren't needed.