SCOOP is a Python software module for distributing concurrent tasks on various environments, from heterogeneous grids of workstations to supercomputers. It uses ØMQ and the package as building blocks to encapsulate and distribute tasks between processesand/or systems. Its interface is inspired from the proposal. SCOOP is targeted towards scientific applications that require execution of many loosely coupled tasks using all available hardware resources. These resources need to be accessible through SSH.
History
SCOOP was initiated by Yannick Hold and Marc Parizeau at the Computer Vision and Systems Laboratory of Université Laval. It is an iterative step over the now deprecated DTM module of the DEAP framework for developing evolutionary algorithm. While DTM used MPI for its communications, SCOOP uses instead ØMQ.
Network topology
SCOOP uses the Broker Architecture to distribute its Futures. It is based on a central element, called the Broker, that dispatches work to its workers. The main difference between this pattern and a Master/slave topology reside in the Future origin. In the Broker architecture, the Futures emanate from a worker, which is located on the periphery of the topology, instead of the master in the Master/slave architecture. This allows higher reliability in regard to worker faults and generally better performances due to the generic function of the Broker. Since he doesn't need to serialize nor deserialize any Future to route them around the grid, his workload consists of networking or interprocess I/O and almost no CPU processing time. This lowers the bottleneck of the Broker topology. The Broker architecture won't stress the networking fabric and elements as much as a totally distributed topology since there is only one connection required on every worker.
Example
An introductory parallel "Hello, world!" example is implemented this way: from scoop import futures def hello_world -> str: return "Hello World from Future #".format if __name__ "__main__": return_values = futures.map print
