PAQ


PAQ is a series of lossless data compression archivers that have gone through collaborative development to top rankings on several benchmarks measuring compression ratio. Specialized versions of PAQ have won the Hutter Prize and the Calgary Challenge. PAQ is free software distributed under the GNU General Public License.

Algorithm

PAQ uses a context mixing algorithm. Context mixing is related to prediction by partial matching in that the compressor is divided into a predictor and an arithmetic coder, but differs in that the next-symbol prediction is computed using a weighted combination of probability estimates from a large number of models conditioned on different contexts. Unlike PPM, a context doesn't need to be contiguous. Most PAQ versions collect next-symbol statistics for the following contexts:
All PAQ versions predict and compress one bit at a time, but differ in the details of the models and how the predictions are combined and postprocessed. Once the next-bit probability is determined, it is encoded by arithmetic coding. There are three methods for combining predictions, depending on the version:
PAQ1SSE and later versions postprocess the prediction using secondary symbol estimation. The combined prediction and a small context are used to look up a new prediction in a table. After the bit is encoded, the table entry is adjusted to reduce the prediction error. SSE stages can be pipelined with different contexts or computed in parallel with the outputs averaged.

Arithmetic coding

A string s is compressed to the shortest byte string representing a base-256 big-endian number x in the range such that P ≤ x < P, where P is the probability that a random string r with the same length as s will be lexicographically less than s. It is always possible to find an x such that the length of x is at most one byte longer than the Shannon limit, −log2P bits. The length of s is stored in the archive header.
The arithmetic coder in PAQ is implemented by maintaining for each prediction a lower and upper bound on x, initially . After each prediction, the current range is split into two parts in proportion to P and P, the probability that the next bit of s will be a 0 or 1 respectively, given the previous bits of s. The next bit is then encoded by selecting the corresponding subrange to be the new range.
The number x is decompressed back to string s by making an identical series of bit predictions. The range is split as with compression. The portion containing x becomes the new range, and the corresponding bit is appended to s.
In PAQ, the lower and upper bounds of the range are represented in 3 parts. The most significant base-256 digits are identical, so they can be written as the leading bytes of x. The next 4 bytes are kept in memory, such that the leading byte is different. The trailing bits are assumed to be all zeros for the lower bound and all ones for the upper bound. Compression is terminated by writing one more byte from the lower bound.

Adaptive model weighting

In PAQ versions through PAQ6, each model maps a set of distinct contexts to a pair of counts,, a count of zero bits, and, a count of 1 bits. In order to favor recent history, half of the count over 2 is discarded when the opposite bit is observed. For example, if the current state associated with a context is and a 1 is observed, then the counts are updated to.
A bit is arithmetically coded with space proportional to its probability, either P or P = 1 − P. The probabilities are computed by weighted addition of the 0 and 1 counts:
where wi is the weight of the i-th model. Through PAQ3, the weights were fixed and set in an ad-hoc manner. Beginning with PAQ4, the weights were adjusted adaptively in the direction that would reduce future errors in the same context set. If the bit to be coded is y, then the weight adjustment is:
Beginning with PAQ7, each model outputs a prediction. These predictions are averaged in the logistic domain:
where P is the probability that the next bit will be a 1, Pi is the probability estimated by the i-th model, and
After each prediction, the model is updated by adjusting the weights to minimize coding cost:
where η is the learning rate, y is the predicted bit, and is the prediction error. The weight update algorithm differs from backpropagation in that the terms PP are dropped. This is because the goal of the neural network is to minimize coding cost, not root mean square error.
Most versions of PAQ use a small context to select among sets of weights for the neural network. Some versions use multiple networks whose outputs are combined with one more network prior to the SSE stages. Furthermore, for each input prediction there may be several inputs which are nonlinear functions of Pi in addition to stretch.

Context modeling

Each model partitions the known bits of s into a set of contexts and maps each context to a bit history represented by an 8-bit state. In versions through PAQ6, the state represents a pair of counters. In PAQ7 and later versions under certain conditions, the state also represents the value of the last bit or the entire sequence. The states are mapped to probabilities using a 256-entry table for each model. After a prediction by the model, the table entry is adjusted slightly to reduce the prediction error.
In all PAQ8 versions, the representable states are as follows:
To keep the number of states to 256, the following limits are placed on the representable counts:,,,,,,,,. If a count exceeds this limit, then the next state is one chosen to have a similar ratio of n0 to n1. Thus, if the current state is and a 1 is observed, then the new state is not. Rather, it is.
Most context models are implemented as hash tables. Some small contexts are implemented as direct lookup tables.

Text preprocessing

Some versions of PAQ, in particular PAsQDa, PAQAR, and PAQ8HP1 through PAQ8HP8 preprocess text files by looking up words in an external dictionary and replacing them with 1- to 3-byte codes. In addition, uppercase letters are encoded with a special character followed by the lowercase letter. In the PAQ8HP series, the dictionary is organized by grouping syntactically and semantically related words together. This allows models to use just the most significant bits of the dictionary codes as context.

Comparison

The following table is a sample from the by Matt Mahoney that consists of a file consisting of 109 bytes of English Wikipedia text.
ProgramCompressed size % of original sizeCompression time Memory
PAQ8HP8133,423,10964 6391849
PPMd183,976,014880256
bzip2254,007,8753798
InfoZIP322,649,7031040.1

See Lossless compression benchmarks for a list of file compression benchmarks.

History

The following lists the major enhancements to the PAQ algorithm. In addition, there have been a large number of incremental improvements, which are omitted.
The series PAQ8HP1 through PAQ8HP8 were released by Alexander Ratushnyak from August 21, 2006 through January 18, 2007 as Hutter Prize submissions. The Hutter Prize is a text compression contest using a 100 MB English and XML data set derived from Wikipedia's source. The PAQ8HP series was forked from PAQ8H. The programs include text preprocessing dictionaries and models tuned specifically to the benchmark. All non-text models were removed. The dictionaries were organized to group syntactically and semantically related words and to group words by common suffix. The former strategy improves compression because related words can be modeled on the high order bits of their dictionary codes. The latter strategy makes the dictionary easier to compress. The size of the decompression program and compressed dictionary is included in the contest ranking.
On October 27, 2006, it was announced that PAQ8HP5 won a Hutter Prize for Lossless Compression of Human Knowledge of €3,416.
On June 30, 2007, Ratushnyak's paq8hp12 was awarded a second Hutter prize of €1732, improving upon his previous record by 3.46%.

PAQ derivations

Being free software, PAQ can be modified and redistributed by anyone who has a copy. This has allowed other authors to fork the PAQ compression engine and add new features such as a graphical user interface or better speed. Notable PAQ derivatives include: