Did anyone use the Optional Dagster type? I am try...
# announcements
f
Did anyone use the Optional Dagster type? I am trying to use it to pass an optional Input (pre-trained scaler) to a solid but it doesn’t seem to be working.
Copy code
@solid(
    input_defs=[
        InputDefinition(name="df_X", dagster_type=DataFrame),
        InputDefinition(name='scaler', dagster_type=Optional[Any])],
    output_defs=[
        OutputDefinition(name="df_scaled_X", dagster_type=DataFrame),
        OutputDefinition(name="scaler", dagster_type=Any)],
)
def min_max_scaler(_, df_X, scaler):
    if not scaler:
        scaler = MinMaxScaler().fit(df_X)
    yield Output(pd.DataFrame(scaler.transform(df_X), columns=df_X.columns), "df_scaled_X")
    yield Output(scaler, "scaler")
And then I call the solid like this :
Copy code
min_max_scaler_train = min_max_scaler.alias("min_max_scaler_train")
df_X_train, scaler = min_max_scaler_train(df_X_train)

min_max_scaler_test = min_max_scaler.alias("min_max_scaler_test")
df_X_test, _ = min_max_scaler_test(df_X_test, scaler)
But then Dagit complains about a missing Input definition for the scaler, in the configuration.
m
hi @Frederic Abiven, we have a concept of inputs that don't need to be satisfied in order for a solid to execute -- the semantics of
Optional
are that the input can have the value
None
, but it must still be provided for the solid to execue
f
This was my understanding too. So I tried the following :
Copy code
min_max_scaler_train = min_max_scaler.alias("min_max_scaler_train")
df_X_train, scaler = min_max_scaler_train(df_X_train, None)
min_max_scaler_test = min_max_scaler.alias("min_max_scaler_test")
df_X_test, _ = min_max_scaler_test(df_X_test, scaler)
But then got the error :
Copy code
In @composite_solid modelling, received invalid type <class 'NoneType'> for input "scaler" (passed by keyword) in solid invocation "min_max_scaler_train". Must pass the output from previous solid invocations or inputs to the composition function as inputs when invoking solids during composition.
d
@Frederic Abiven may be worth updating your input definition to:
InputDefinition(name='scaler', dagster_type=Optional[Any], default_value=None)
and see if that behaves the way you expect it to
1
I think that way should allow you to not supply the
None
value at all
f
Amazing! It works. Thank you
🙌 1