In numerical analysis, pairwise summation, also called cascade summation, is a technique to sum a sequence of finite-precision floating-point numbers that substantially reduces the accumulated round-off error compared to naively accumulating the sum in sequence. Although there are other techniques such as Kahan summation that typically have even smaller round-off errors, pairwise summation is nearly as good while having much lower computational cost—it can be implemented so as to have nearly the same cost as naive summation. In particular, pairwise summation of a sequence of n numbers xn works by recursively breaking the sequence into two halves, summing each half, and adding the two sums: a divide and conquer algorithm. Its worst-case roundoff errors grow asymptotically as at most O, where ε is the machine precision. In comparison, the naive technique of accumulating the sum in sequence has roundoff errors that grow at worst as O. Kahan summation has a worst-case error of roughly O, independent of n, but requires several times more arithmetic operations. If the roundoff errors are random, and in particular have random signs, then they form a random walk and the error growth is reduced to an average of for pairwise summation. A very similar recursive structure of summation is found in many fast Fourier transform algorithms, and is responsible for the same slow roundoff accumulation of those FFTs.
The algorithm
In pseudocode, the pairwise summation algorithm for an array of length > 0 can be written: s = pairwise ifn ≤ Nbase case: naive summation for a sufficiently small array s = x fori = 2 to n s = s + x elsedivide and conquer: recursively sum two halves of the array m = floor s = pairwise + pairwise end if For some sufficiently small, this algorithm switches to a naive loop-based summation as a base case, whose error bound is O. The entire sum has a worst-case error that grows asymptotically as O for large n, for a given condition number. In an algorithm of this sort, it is desirable to use a larger base case in order to amortize the overhead of the recursion. If N = 1, then there is roughly one recursive subroutine call for every input, but more generally there is one recursive call for every N/2 inputs if the recursion stops at exactly n = N. By making Nsufficiently large, the overhead of recursion can be made negligible. Regardless of N, exactly n−1 additions are performed in total, the same as for naive summation, so if the recursion overhead is made negligible then pairwise summation has essentially the same computational cost as for naive summation. A variation on this idea is to break the sum into b blocks at each recursive stage, summing each block recursively, and then summing the results, which was dubbed a "superblock" algorithm by its proposers. The above pairwise algorithm corresponds to b = 2 for every stage except for the last stage which is b = N.
Accuracy
Suppose that one is summing n values xi, for i = 1, ..., n. The exact sum is: . With pairwise summation for a base case N = 1, one instead obtains, where the error is bounded above by: where ε is the machine precision of the arithmetic being employed. Usually, the quantity of interest is the relative error, which is therefore bounded above by: In the expression for the relative error bound, the fraction is the condition number of the summation problem. Essentially, the condition number represents the intrinsic sensitivity of the summation problem to errors, regardless of how it is computed. The relative error bound of everysummation method by a fixed algorithm in fixed precision, is proportional to this condition number. An ill-conditioned summation problem is one in which this ratio is large, and in this case even pairwise summation can have a large relative error. For example, if the summands xi are uncorrelated random numbers with zero mean, the sum is a random walk and the condition number will grow proportional to. On the other hand, for random inputs with nonzero mean the condition number asymptotes to a finite constant as. If the inputs are all non-negative, then the condition number is 1. Note that the denominator is effectively 1 in practice, since is much smaller than 1 until n becomes of order 21/ε, which is roughly 101015 in double precision. In comparison, the relative error bound for naive summation grows as multiplied by the condition number. In practice, it is much more likely that the rounding errors have a random sign, with zero mean, so that they form a random walk; in this case, naive summation has a root mean square relative error that grows as and pairwise summation has an error that grows as on average.
Software implementations
Pairwise summation is the default summation algorithm in NumPy and the Julia technical-computing language, where in both cases it was found to have comparable speed to naive summation. Other software implementations include the HPCsharp library for the C Sharp language.
