Suppose that within a pipeline, there's a composite consisting of a series of solids that share data using a local filesystem (because there's too much data to share over the network). As such, I need to ensure that all solids within that composite execute on the same machine. Every other solid in the pipeline, though, does not have any such constraints, and can be freely distributed via a celery executor or whatever.
Is there a way of describing this in dagster today?
Or do I need to treat the composite as a separate dag run?