https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Chaitya

03/08/2023, 9:25 PM
Is there a way to implement an op that is also a classmethod or staticmethod of an abstract class in python? I've got an abstract class that i'd want my fellow developers to take advantage of, so they can more easily create dagster ops within our framework, but these classes are singletons and not meant to be instantiated... Any attempts to use
@classmethod
are thwarted by dagster's argument inference from the function signature.
z

Zach P

03/08/2023, 9:40 PM
What are you trying to do here? There’s examples in the docs of using op factories, which could potentially be helpful for you (EG: Asset factories may help). Alternatively, you could create dagster resources that contain the shared behavior you need without abstract classes and such.
c

Chaitya

03/08/2023, 9:45 PM
I'd like to use an abstract class to encapsulate a lot of the per-node utility functions that I expect my developers to need when defining their dagster ops. For example:
Copy code
class AbstractNode(ABC):
    @op()
    @classmethod
    def run(cls, context):
       # user code here in derived classes
       raise NotImplementedError
        
    @classmethod
    def some_utility(): 
       ...

    
class MyNode(AbstractNode):
    @op()
    @classmethod
    def run(cls, context):
        return some_complex_logic(cls.some_utility(context.op_config["my_param"]))
Ideally, I can set up a nice framework within the abstract class that lets me set requirements for how all subclasses'
run
ops are defined
z

Zach P

03/08/2023, 10:01 PM
What sort of per-node ops are you thinking about? Also keep in mind that for many things you should be using assets, not ops. If you want to share utility functions, I’d suggest creating a dagster resource to share. If you really want the ability to user superclasses to define/restrict behavior, you could potentially write your framework around factories that produce op or asset definitions. Finally, if you still want subclasses and stuff for the assets/ops themselves, you could potentailly try mixins.
Copy code
In [16]: class AbstractAssetMixin(AssetsDefinition, ABC):
    ...:     def __init__(self, *args, **kwargs):
    ...:         x = True
    ...: 

In [17]: class MyTeamMixin(AbstractAssetMixin):
    ...:     def must_implement(self):
    ...:         return "hello world"
    ...: 

In [18]: x = MyTeamMixin()

In [19]: isinstance(x, AssetsDefinition)
Out[19]: True
I havent tested it anymore than this, so it may not actually work.
c

chris

03/08/2023, 10:21 PM
We had considered supporting a syntax similar to this at one point, but never really went anywhere. Can perhaps still use an abstract class as an “Op builder” if that makes sense, and then programmatically generate all of the software artifacts. A pattern more like this:
Copy code
class AbstractNode(ABC):
    @classmethod
    def run(cls, context):
       # user code here in derived classes
       raise NotImplementedError
        
    @classmethod
    def some_utility(): 
       ...

    @classmethod
    def generate_op(cls) -> OpDefinition:
        @op
        def the_op():
            util = cls.some_utility()
            cls.run(util)
        return the_op
c

Chaitya

03/08/2023, 10:30 PM
@chris that looks nice -- Let me give it a shot. will report back.
That usage pattern seems to be working for the functionality i'm looking for! Thanks a ton!
c

chris

03/08/2023, 11:33 PM
awesome blob salute