solids and other constructs in dagster are mostly functions, can solid be classes that extends dagster's constructs?

Hey @osayamen omigie, while many constructs in Dagster are defined using decorators and functions, these decorators actually always construct a

Definition

class. For example, the

@solid

decorator constructs a

SolidDefinition

, and

@pipeline

constructs a

PipelineDefinition

. You can use these definition classes directly in place of the decorator + function, which is especially useful when programmatically generating them, and these classes can be extended.

I have seen the definition classes. i am yet to see how it helps me. and yes i do want to instantiate solids programmatically. My use-case: am trying to adapt a system we currently have where we use the concepts of plugins that are composed together at runtime based on a config file. we have a pipeline-like application that looks at a config file and based of the list of plugins in the config file it goes to the plugin folder and instantiates them and runs them in order they were listed in the config file, passing a context-like object to the next plugin i think we can use dagster to replace this system and i am working on some sort of POC and am trying to do it without a lot of changes to the current plugin classes. if i could extended them of solids and have minimal changes. i guess in summary i am looking to compose a pipeline DAG of solids at runtime choosing the solids from a repertoire of solids based on the solids chosen in a config file

heres one example of dynamic pipeline generation from a config file https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/dep_dsl/pipeline.py