PyOperator (http://pchanial.github.com/pyoperators) is a scalable package that provides a generic and efficient set of operators and solvers for high-performance computing. These operators are multi-dimensional functions that, if linear, behave like matrices with a sparse storage footprint. Operators can be transposed, added, multiplied and composed in a natural way so that they can be used as the building blocks or the subsystems of a complex instrument acquisition model. The chaining of the operations has been optimised to minimise the number of memory allocations, the memory footprint and to maximise the cache locality by relying on properties such as whether or not the operator can perform in-place operations on its input or on its output. The Python overhead has been reduced to a mostly negligeable fraction. Rules can be attached to operators, to trigger algebraic simplifications, which leads to additional speedups. The package also makes it very easy to manipulate operators to form block diagonal, block column and block row operators.
We will then present real-world applications of the PyOperators tools in non-distributed and distributed (using MPI) architectures. The scalability has so far been demonstrated using up to thousands of cores. This package, coupled with the Tamasis project, has successfully been used to construct fast simulators of astrophysical instruments and to solve the inverse problem related to the acquisition of the signal through these simulators, resulting in solving systems of equations with up to 10^9 unknowns. In particular, we will present results for the PACS photometer on-board the Herschel satellite, and the MIRIM instrument on-board JWST.
Pierre Chanial & Nicolas Barbey