Hi all I want to use a `dagster type loader` to map a string dagster #announcements

Hi all. I want to use a `dagster_type_loader` to ...

Ryan

02/02/2021, 3:29 PM

Hi all. I want to use a

dagster_type_loader

to map a string value from the config to a domain Enum, using the basic

MyEnum[choice_name]

. I was expecting that any errors thrown from type loader functions (in this case a

KeyError

whenever an invalid enum name is specified) would result in a "Type check failed" event, but I see that the exception just propagates and breaks the pipeline at that point. Is there anything I can do to have invalid inputs to type loader functions handled slightly more gracefully within the framework? Something like:

Copy code

@dagster_type_loader(str)
def load_my_enum(_, choice_name: str):
    try:
       return MyEnum[choice_name]
    except KeyError as ex:
        raise DagsterSpeciallyHandledError()

I tried with

DagsterInvalidConfigError

but it didn't seem ergonomic to use from this context.

alex

02/02/2021, 4:11 PM

@dagster_type_loader(str)

instead of taking an arbitrary

str

in config space, you could define an enum

Copy code

from dagster import Enum

@dagster_type_loader(Enum.from_python_enum(MyEnum))

alex

02/02/2021, 4:12 PM

this would cause invalid values to fail at config validation time

alex

02/02/2021, 4:12 PM

https://docs.dagster.io/_apidocs/config#dagster.Enum

sandy

02/02/2021, 4:26 PM

@Ryan - when you say you'd like the error to be handled gracefully, what behavior would you ideally like to happen following the invalid input? E.g. would you like the pipeline to be able to proceed with some sort of fallback value?

Ryan

02/02/2021, 6:02 PM

Fab, thanks a lot @alex, that's got me where I wanted to be:

Copy code

from dagster import Enum, dagster_type_loader

DagsterMyEnum = Enum.from_python_enum(MyEnum)

@dagster_type_loader(DagsterMyEnum)
def load_my_enum(_, choice: DagsterMyEnum):
    return MyEnum[choice.name]

This way, in my parameterised tests, I can consistently get a

DagsterInvalidConfigError

whether I pass it a string key that's not valid, or another data type altogether 👍🏼

Ryan

02/02/2021, 6:13 PM

Hey @sandy 🙂 It's basically a question of the API of

dagster_type_loader

. It wasn't necessarily that the pipeline should be able to proceed, just that I'd like to be able to express to the pipeline (I was thinking via some agreed exception type) that the conversion from the config value to my target type has failed in a way that Dagster understands, e.g.

InvalidConfig

. Then, by "graceful" I just meant that the pipeline would actually be able to finish as it normally would with an invalid input (i.e. early) and then I'd be able to check the return value of

execute_solid

to investigate what happened. Of course if a non-Dagster exception bubbles up then fine to explode immediately 🧨

Ryan

02/02/2021, 6:13 PM

Hope that makes sense!

sandy

02/02/2021, 7:30 PM

Thanks Ryan - that makes total sense

2 Views

Open in Slack

Previous Next