https://dagster.io/ logo
Powered by
r

Ryan

02/02/2021, 3:29 PM
Hi all. I want to use a 
dagster_type_loader
to map a string value from the config to a domain Enum, using the basic 
MyEnum[choice_name]
. I was expecting that any errors thrown from type loader functions (in this case a 
KeyError
whenever an invalid enum name is specified) would result in a "Type check failed" event, but I see that the exception just propagates and breaks the pipeline at that point. Is there anything I can do to have invalid inputs to type loader functions handled slightly more gracefully within the framework? Something like:
Copy code
@dagster_type_loader(str)
def load_my_enum(_, choice_name: str):
    try:
       return MyEnum[choice_name]
    except KeyError as ex:
        raise DagsterSpeciallyHandledError()
I tried with 
DagsterInvalidConfigError
but it didn't seem ergonomic to use from this context.
a

alex

02/02/2021, 4:11 PM
@dagster_type_loader(str)
instead of taking an arbitrary 
str
in config space, you could define an enum
Copy code
from dagster import Enum

@dagster_type_loader(Enum.from_python_enum(MyEnum))
this would cause invalid values to fail at config validation time
s

sandy

02/02/2021, 4:26 PM
@Ryan - when you say you'd like the error to be handled gracefully, what behavior would you ideally like to happen following the invalid input? E.g. would you like the pipeline to be able to proceed with some sort of fallback value?
r

Ryan

02/02/2021, 6:02 PM
Fab, thanks a lot @alex, that's got me where I wanted to be:
Copy code
from dagster import Enum, dagster_type_loader

DagsterMyEnum = Enum.from_python_enum(MyEnum)

@dagster_type_loader(DagsterMyEnum)
def load_my_enum(_, choice: DagsterMyEnum):
    return MyEnum[choice.name]
This way, in my parameterised tests, I can consistently get a 
DagsterInvalidConfigError
whether I pass it a string key that's not valid, or another data type altogether 👍🏼
Hey @sandy 🙂 It's basically a question of the API of 
dagster_type_loader
. It wasn't necessarily that the pipeline should be able to proceed, just that I'd like to be able to express to the pipeline (I was thinking via some agreed exception type) that the conversion from the config value to my target type has failed in a way that Dagster understands, e.g. 
InvalidConfig
. Then, by "graceful" I just meant that the pipeline would actually be able to finish as it normally would with an invalid input (i.e. early) and then I'd be able to check the return value of 
execute_solid
to investigate what happened. Of course if a non-Dagster exception bubbles up then fine to explode immediately 🧨
Hope that makes sense!
s

sandy

02/02/2021, 7:30 PM
Thanks Ryan - that makes total sense
2 Views
Powered by