SHA-3


SHA-3 is the latest member of the Secure Hash Algorithm family of standards, released by NIST on August 5, 2015. Although part of the same series of standards, SHA-3 is internally different from the MD5-like structure of SHA-1 and SHA-2.
SHA-3 is a subset of the broader cryptographic primitive family Keccak, designed by Guido Bertoni, Joan Daemen, Michaël Peeters, and Gilles Van Assche, building upon RadioGatún. Keccak's authors have proposed additional uses for the function, not standardized by NIST, including a stream cipher, an authenticated encryption system, a "tree" hashing scheme for faster hashing on certain architectures, and AEAD ciphers Keyak and Ketje.
Keccak is based on a novel approach called sponge construction. Sponge construction is based on a wide random function or random permutation, and allows inputting any amount of data, and outputting any amount of data, while acting as a pseudorandom function with regard to all previous inputs. This leads to great flexibility.
NIST does not currently plan to withdraw SHA-2 or remove it from the revised Secure Hash Standard. The purpose of SHA-3 is that it can be
directly substituted for SHA-2 in current applications if necessary, and to significantly improve the robustness of NIST's overall hash algorithm toolkit.
The creators of the Keccak algorithms and the SHA-3 functions suggest using the faster function KangarooTwelve with adjusted parameters and a new tree hashing mode without extra overhead for small message sizes.

History

The Keccak algorithm is the work of Guido Bertoni, Joan Daemen, Michael Peeters, and Gilles Van Assche. It is based on earlier hash function designs PANAMA and RadioGatún. PANAMA was designed by Daemen and Craig Clapp in 1998. RadioGatún, a successor of PANAMA, was designed by Daemen, Peeters, and Van Assche, and was presented at the NIST Hash Workshop in 2006. The reference implementation source code was dedicated to public domain via CC0 waiver.
In 2006 NIST started to organize the NIST hash function competition to create a new hash standard, SHA-3. SHA-3 is not meant to replace SHA-2, as no significant attack on SHA-2 has been demonstrated. Because of the successful attacks on MD5, SHA-0 and SHA-1,
NIST perceived a need for an alternative, dissimilar cryptographic hash, which became SHA-3.
After a setup period, admissions were to be submitted by the end of 2008. Keccak was accepted as one of the 51 candidates. In July 2009, 14 algorithms were selected for the second round. Keccak advanced to the last round in December 2010.
During the competition, entrants were permitted to "tweak" their algorithms to address issues that were discovered. Changes that have been made to Keccak are:
On October 2, 2012, Keccak was selected as the winner of the competition.
In 2014, the NIST published a draft FIPS 202 "SHA-3 Standard: Permutation-Based Hash and Extendable-Output Functions". FIPS 202 was approved on August 5, 2015.
On August 5, 2015 NIST announced that SHA-3 had become a hashing standard.

Design

SHA-3 uses the sponge construction, in which data is "absorbed" into the sponge, then the result is "squeezed" out. In the absorbing phase, message blocks are XORed into a subset of the state, which is then transformed as a whole using a permutation function. In the "squeeze" phase, output blocks are read from the same subset of the state, alternated with the state transformation function. The size of the part of the state that is written and read is called the "rate", and the size of the part that is untouched by input/output is called the "capacity". The capacity determines the security of the scheme. The maximum security level is half the capacity.
Given an input bit string, a padding function, a permutation function that operates on bit blocks of width, a rate and an output length, we have capacity and the sponge construction, yielding a bit string of length, works as follows:
The fact that the internal state S contains c additional bits of information in addition to what is output to Z prevents the length extension attacks that SHA-2, SHA-1, MD5 and other hashes based on the Merkle–Damgård construction are susceptible to.
In SHA-3, the state S consists of a array of w-bit words, b = 5 × 5 × w = 5 × 5 × 64 = 1600 bits total. Keccak is also defined for smaller power-of-2 word sizes w down to 1 bit. Small state sizes can be used to test cryptanalytic attacks, and intermediate state sizes can be used in practical, lightweight applications.
For SHA-3-224, SHA-3-256, SHA-3-384, and SHA-3-512 instances, r is greater than d, so there is no need for additional block permutations in the squeezing phase; the leading d bits of the state are the desired hash. However, SHAKE-128 and SHAKE-256 allow an arbitrary output length, which is useful in applications such as optimal asymmetric encryption padding.

Padding

To ensure the message can be evenly divided into r-bit blocks, padding is required. SHA-3 uses the pattern 10*1 in its padding function: a 1 bit, followed by zero or more 0 bits and a final 1 bit.
The maximum of zero bits occurs when the last message block is bits long. Then another block is added after the initial 1 bit, containing zero bits before the final 1 bit.
The two 1 bits will be added even if the length of the message is already divisible by r. In this case, another block is added to the message, containing a 1 bit, followed by a block of zero bits and another 1 bit. This is necessary so that a message with length divisible by r ending in something that looks like padding does not produce the same hash as the message with those bits removed.
The initial 1 bit is required so messages differing only in a few additional 0 bits at the end do not produce the same hash.
The position of the final 1 bit indicates which rate r was used, which is required for the security proof to work for different hash variants. Without it, different hash variants of the same short message would be the same up to truncation.

The block permutation

The block transformation f, which is Keccak-f for SHA-3, is a permutation that uses XOR, AND and NOT operations, and is designed for easy implementation in both software and hardware.
It is defined for any power-of-two word size, bits. The main SHA-3 submission uses 64-bit words,.
The state can be considered to be a array of bits. Let be bit of the input, using a little-endian bit numbering convention and row-major indexing. I.e. selects the row, the column, and the bit.
Index arithmetic is performed modulo 5 for the first two dimensions and modulo w for the third.
The basic block permutation function consists of rounds of five steps:
;
;
;
;
;

Speed

The speed of SHA-3 hashing of long messages is dominated by the computation of f = Keccak-f and XORing S with the extended Pi, an operation on b = 1600 bits. However, since the last c bits of the extended Pi are 0 anyway, and XOR with 0 is a NOP, it is sufficient to perform XOR operations only for r bits. The lower r is, the less efficient but more secure the hashing becomes since fewer bits of the message can be XORed into the state before each application of the computationally expensive f.
The authors report the following speeds for software implementations of Keccak-f plus XORing 1024 bits, which roughly corresponds to SHA3-256:
For the exact SHA3-256 on x86-64, Bernstein measures 11.7–12.25 cpb depending on the CPU. SHA-3 has been criticized for being slow on instruction set architectures which do not have instructions meant specially for computing Keccak functions faster – SHA2-512 is more than twice as fast as SHA3-512, and SHA-1 is more than three times as fast on an Intel Skylake processor clocked at 3.2 GHz. The authors have reacted to this criticism by suggesting to use SHAKE128 and SHAKE256 instead of SHA3-256 and SHA3-512, at the expense of cutting the preimage resistance in half. With this, performance is on par with SHA2-256 and SHA2-512.
However, in hardware implementations, SHA-3 is notably faster than all other finalists, and also faster than SHA-2 and SHA-1.
ARM's ARMv8 and IBM's s390x architectures already include special instructions which enable Keccak algorithms to execute faster.

Instances

The NIST standard defines the following instances, for message M and output length d:
With the following definitions
Note that the appended postfixes are written as bit strings, not hexadecimal digits.
The SHA-3 instances are the drop-in replacements for SHA-2, with identical security claims. SHAKE instances are so called XOF's, Extendable Output Functions. For example, SHAKE128 can be used as a hash function with a 256-bit length and 128-bit overall security.
Note that all instances append some bits to the message, the rightmost of which represent the domain separation suffix. The purpose of this is to ensure that it is not possible to construct messages that produce the same hash output for different applications of the Keccak hash function. The following domain separation suffixes exist:
SuffixMeaning
...0reserved for future use
01SHA-3
...11RawSHAKE

RawSHAKE is the basis for the Sakura coding for tree hashing, which has not been standardized yet. However, the SHAKE suffix has been carefully chosen so that it is forward compatible with Sakura. Sakura appends 0 for a chaining hop or 1 for a message, then 10*0 for a non-final node or 1 for a final node, before it applies RawSHAKE. Sequential hashing corresponds to a hop tree with a single message node, which means that 11 is appended to the message before RawSHAKE is applied. Thus, the SHAKE XOFs append 1111 to the message, i.e., 1 for message, 1 for final node, and 11 for the RawSHAKE domain separation suffix.
Since 10*1 padding always adds at least two bits, in byte-aligned libraries there are always six unused zero bits. Therefore, these appended extra bits never make the padded message longer.

Additional instances

In December 2016 NIST published a new document, NIST SP.800-185, describing additional SHA-3 derived functions:
InstanceDescription
cSHAKE128A version of SHAKE supporting explicit domain separation via customization parameters.
cSHAKE256A version of SHAKE supporting explicit domain separation via customization parameters.
KMAC128A keyed hash function based on Keccak. Can also be used without a key as a regular hash function.
KMAC256A keyed hash function based on Keccak. Can also be used without a key as a regular hash function.
KMACXOF128A keyed hash function based on Keccak. Can also be used without a key as a regular hash function.
KMACXOF256A keyed hash function based on Keccak. Can also be used without a key as a regular hash function.
TupleHash128A function for hashing tuples of strings. The output of this function depends on both the contents and the sequence of input strings.
TupleHash256A function for hashing tuples of strings. The output of this function depends on both the contents and the sequence of input strings.
TupleHashXOF128A function for hashing tuples of strings. The output of this function depends on both the contents and the sequence of input strings.
TupleHashXOF256A function for hashing tuples of strings. The output of this function depends on both the contents and the sequence of input strings.
ParallelHash128A function designed to exploit parallelism in modern processors for faster hashing. Unlike KangarooTwelve, does not use reduced-round Keccak.
ParallelHash256A function designed to exploit parallelism in modern processors for faster hashing. Unlike KangarooTwelve, does not use reduced-round Keccak.
ParallelHashXOF128A function designed to exploit parallelism in modern processors for faster hashing. Unlike KangarooTwelve, does not use reduced-round Keccak.
ParallelHashXOF256A function designed to exploit parallelism in modern processors for faster hashing. Unlike KangarooTwelve, does not use reduced-round Keccak.

• X is the main input bit string. It may be of any length, including zero.
• L is an integer representing the requested output length in bits.
• N is a function-name bit string, used by NIST to define functions based on cSHAKE. When no function other than cSHAKE is desired, N is set to the empty string.
• S is a customization bit string. The user selects this string to define a variant of the function. When no customization is desired, S is set to the empty string.
• K is a key bit string of any length, including zero.
• B is the block size in bytes for parallel hashing. It may be any integer such that 0 < B < 22040.

Later developments

Tree Hashing

In 2016 the same team that made the SHA-3 functions and the Keccak algorithm introduced faster reduced-rounds alternatives which can exploit the availability of parallel execution because of using tree hashing: KangarooTwelve and MarsupilamiFourteen.
These functions differ from ParallelHash, the FIPS standardized Keccak-based parallelizable hash function, with regard to the parallelism, in that they are faster than ParallelHash for small message sizes.
The reduced number of rounds is justified by the huge cryptanalytic effort focused on Keccak which did not produce practical attacks on anything close to twelve-round Keccak. These higher-speed algorithms are not part of SHA-3, and thus are not FIPS compliant; but they are just as secure as the SHA-3 functions, because they use the same Keccak permutation and there are no attacks on 12-round Keccak.
KangarooTwelve is a higher-performance reduced-round version of Keccak which claims to have 128 bits of security while having performance as high as 0.55 cycles per byte on a Skylake CPU. This algorithm is an IETF RFC draft.
MarsupilamiFourteen, a slight variation on KangarooTwelve, uses 14 rounds of the Keccak permutation and claims 256 bits of security. Note that 256-bit security is not more useful in practice than 128-bit security, but may be required by some standards. 128 bits are already sufficient to defeat brute-force attacks on current hardware, so having 256-bit security does not add practical value, unless the user is worried about significant advancements in the speed of classical computers. For resistance against quantum computers, see below.
KangarooTwelve and MarsupilamiFourteen are Extendable-Output Functions, similar to SHAKE, therefore they generate closely related output for a common message with different output length. Such property is not exhibited by hash functions such as SHA-3 or ParallelHash.

The Farfalle Construction

In 2016, the Keccak team released a different construction called Farfalle construction, and Kravatte, an instance of Farfalle using the Keccak-p permutation., as well as two authenticated encryption algorithms Kravatte-SANE and Kravatte-SANSE

Security against quantum attacks

There is a general result that quantum computers can perform a structured preimage attack in, while a classical brute-force attack needs 2d. A structured preimage attack implies a second preimage attack and thus a collision attack. A quantum computer can also perform a birthday attack, thus break collision resistance, in . Noting that the maximum strength can be, this gives the following upper bounds on the quantum security of SHA-3:
It has been shown that the Merkle–Damgård construction, as used by SHA-2, is collapsing and, by consequence, quantum collision-resistant, but for the sponge construction used by SHA-3, the authors provide proofs only for the case that the block function f is not efficiently invertible; Keccak-f, however, is efficiently invertible, and so their proof does not apply.

Capacity change controversy

In February 2013 at the RSA Conference, and then in August 2013 at CHES, NIST announced they would select different values for the capacity, i.e. the security parameter, for the SHA-3 standard, compared to the submission. The changes caused some turmoil.
The hash function competition called for hash functions at least as secure as the SHA-2 instances. It means that a d-bit output should have d/2-bit resistance to collision attacks and d-bit resistance to preimage attacks, the maximum achievable for d bits of output. Keccak's security proof allows an adjustable level of security based on a "capacity" c, providing c/2-bit resistance to both collision and preimage attacks. To meet the original competition rules, Keccak's authors proposed c=2d. The announced change was to accept the same d/2-bit security for all forms of attack and standardize c=d. This would have sped up Keccak by allowing an additional d bits of input to be hashed each iteration. However, the hash functions would not have been drop-in replacements with the same preimage resistance as SHA-2 anymore; it would have been cut in half, making it vulnerable to advances in quantum computing, which effectively would cut it in half once more.
In September 2013, Daniel J. Bernstein suggested on the NIST hash-forum mailing list to strengthen the security to the 576-bit capacity that was originally proposed as the default Keccak, in addition to and not included in the SHA-3 specifications. This would have provided at least a SHA3-224 and SHA3-256 with the same preimage resistance as their SHA-2 predecessors, but SHA3-384 and SHA3-512 would have had significantly less preimage resistance than their SHA-2 predecessors. In late September, the Keccak team responded by stating that they had proposed 128-bit security by setting as an option already in their SHA-3 proposal. Although the reduced capacity was justifiable in their opinion, in the light of the negative response, they proposed raising the capacity to c = 512 bits for all instances. This would be as much as any previous standard up to the 256-bit security level, while providing reasonable efficiency, but not the 384-/512-bit preimage resistance offered by SHA2-384 and SHA2-512. The authors tried to justify that with the claim that "claiming or relying on security strength levels above 256 bits is meaningless".
In early October 2013, Bruce Schneier criticized NIST's decision on the basis of its possible detrimental effects on the acceptance of the algorithm, saying:
Paul Crowley, a cryptographer and senior developer at an independent software development company, expressed his support of the decision, saying that Keccak is supposed to be tunable and there is no reason for different security levels within one primitive. He also added:
There was also some confusion that internal changes were made to Keccak. The Keccak team clarified this, stating that NIST's proposal for SHA-3 is a subset of the Keccak family, for which one can generate test vectors using their reference code submitted to the contest, and that this proposal was the result of a series of discussions between them and the NIST hash team. Also, Bruce Schneier corrected his earlier statement, saying:
In response to the controversy, in November 2013 John Kelsey of NIST proposed to go back to the original proposal for all SHA-2 drop-in replacement instances. The reversion was confirmed in the April 2014 draft. This proposal was implemented in the final release standard in August 2015.
The reduced-capacity forms were published as SHAKE128 and SHAKE256, where the number indicates the security level and the number of bits of output is variable, but should be twice as large as the required collision resistance.

Examples of SHA-3 variants

The following hash values are from NIST.gov:
SHA3-224
6b4e03423667dbb73b6e15454f0eb1abd4597f9a1b078e3f5b5a6bc7
SHA3-256
a7ffc6f8bf1ed76651c14756a061d662f580ff4de43b49fa82d80a4b80f8434a
SHA3-384
0c63a75b845e4f7d01107d852e4c2485c51a50aaaa94fc61995e71bbee983a2ac3713831264adb47fb6bd1e058d5f004
SHA3-512
a69f73cca23a9ac5c8b567dc185a756e97c982164fe25859e0d1dcc1475c80a615b2123af1f5f94c11e3e9402c3ac558f500199d95b6d3e301758586281dcd26
SHAKE128
7f9c2ba4e88f827d616045507605853ed73b8093f6efbc88eb1a6eacfa66ef26
SHAKE256
46b9dd2b0ba88d13233b3feb743eeb243fcd52ea62b81b82b50c27646ed5762fd75dc4ddd8c0f200cb05019d67b592f6fc821c49479ab48640292eacb3b7c4be
Changing a single bit causes each bit in the output to change with 50% probability, demonstrating an avalanche effect:
SHAKE128
f4202e3c5852f9182a0430fd8144f0a74b95e7417ecae17db0f8cfeed0e3e66e
SHAKE128
853f4538be0db9621a6cea659a06c1107b1f83f02b13d18297bd39d7411cf10c

Comparison of SHA functions

In the table below, internal state means the number of bits that are carried over to the next block.
Optimized implementation using AVX-512VL of SHA3-256 do achieve about 6.4 cycles per byte for large messages, and about 7.8 cycles per byte when using AVX2 on Skylake CPUs.. Performance on other x86, Power and ARM CPUs depending on instructions used, and exact CPU model varies from about 8 to 15 cycles per byte
, with some older x86 CPUs up to 25-40 cycles per byte.

Implementations

Below is a list of cryptography libraries that support SHA-3:
ARMv8 six-core SoC CPU cores have support for accelerating SHA-3 using specialized instructions from ARMv8.2-SHA crypto extension set.
Some software libraries use vectorization facilities of CPUs to accelerate usage of SHA-3. For example Crypto++ can use SSE2 on x86 for accelerating SHA3, and OpenSSL can use MMX, AVX-512 or AVX-512VL on many x86 systems too.. Also POWER8 CPUs implement 2x64-bit vector rotate, defined in PowerISA 2.07, which can accelerate SHA-3 implementations somehow. Most implementations for ARM doesn't use Neon vector instructions as it is slower than a scalar code, however it can be accelerated using SVE and SVE2 vector instructions.

Usage in protocols

*