Hi, I'm currently playing with Dagster at the moment. Not sure if this is a good use case but would you recommend Dagster for stitching a bunch of Python functions containing some API calls, dataframe methods and string analysis? These functions are short running (ms to few secs) and there maybe 20-50 functions that can be called in different sequences so the pipelines will have to be created dynamically. Also another thing is does it make sense to just deploy Dagster as a python module to check and execute pipelines if there is no need for the dagit and dagster daemon to keep track of runs? Thanks in advance.

so the pipelines will have to be created dynamically

Can you expand on this? Outside of specific dynamic map & collect features, dagster operates on a static pipeline structure once execution begins. We have examples of creating pipelines “dynamically” but they are locked in once execution starts.

does it make sense to just deploy Dagster as a python module to check and execute pipelines if there is no need for the dagit

yea that should work just fine

Can you expand on this?

Thanks. As for dynamic pipeline creation, this is having to create a dynamic number branches out from certain solids, then be collected later. I guess another way to do it instead is just create all the branches beforehand and just yield the specific outputs depending on some conditions. They won't change once they are executed.

yea that should work just fine

cool. It just not able to utilise dagster's daemon functions but more of just using API's for pipeline and solid creation.

if each downstream branch is equivalent, https://docs.dagster.io/_apidocs/dynamic#dynamic-mapping-collect may be relevant