Hi, what are the options for executing solids in a pipeline in parallel? Is it possible to declare sets of solids which should be executed in parallel?
09/28/2020, 2:14 PM
hi richard, some executors support parallel execution and some don't -- so, for instance, by default you can use the multiprocessing executor by adding a chunk of config like
the executors will determine what can be executed in parallel by looking at the data dependencies between solids
so any solids whose dependencies are satisfied can be executed in parallel
if you need to explicitly sequence solids but can't pass data between them for some reason, you can use inputs/outputs with the type
09/28/2020, 2:55 PM
Thanks max that makes sense. Just reading through the Executor docs now. We have solids that might look like they can run parallel but actually need to be run in sequence. Going by the docs I think we could write an executor that would suit our needs.