The goal of steganalysis is to identify suspected packages, determine whether or not they have a payload encoded into them, and, if possible, recover that payload. Unlike cryptanalysis, in which intercepted data contains a message, steganalysis generally starts with a pile of suspect data files, but little information about which of the files, if any, contain a payload. The steganalyst is usually something of a forensic statistician, and must start by reducing this set of data files to the subset most likely to have been altered.
Basic techniques
The problem is generally handled with statistical analysis. A set of unmodified files of the same type, and ideally from the same source as the set being inspected, are analyzed for various statistics. Some of these are as simple as spectrum analysis, but since most image and audio files these days are compressed with lossy compression algorithms, such as JPEG and MP3, they also attempt to look for inconsistencies in the way this data has been compressed. For example, a common artifact in JPEG compression is "edge ringing", where high-frequency components distort neighboring pixels. This distortion is predictable, and simple steganographic encoding algorithms will produce artifacts that are detectably unlikely. One case where detection of suspect files is straightforward is when the original, unmodified carrier is available for comparison. Comparing the package against the original file will yield the differences caused by encoding the payload—and, thus, the payload can be extracted.
In some cases, such as when only a single image is available, more complicated analysis techniques may be required. In general, steganography attempts to make distortion to the carrier indistinguishable from the carrier's noise floor. In practice, however, this is often improperly simplified to deciding to make the modifications to the carrier resemble white noise as closely as possible, rather than analyzing, modeling, and then consistently emulating the actual noise characteristics of the carrier. In particular, many simple steganographic systems simply modify the least-significant bit of a sample; this causes the modified samples to have not only different noise profiles than unmodified samples, but also for their LSBs to have different noise profiles than could be expected from analysis of their higher-order bits, which will still show some amount of noise. Such LSB-only modification can be detected with appropriate algorithms, in some cases detecting encoding densities as low as 1% with reasonable reliability.
Further complications
Encrypted payloads
Detecting a probable steganographic payload is often only part of the problem, as the payload may have been encrypted first. Encrypting the payload is not always done solely to make recovery of the payload more difficult. Most strong ciphers have the desirable property of making the payload appear indistinguishable from uniformly-distributed noise, which can make detection efforts more difficult, and save the steganographic encoding technique the trouble of having to distribute the signal energy evenly.
Barrage noise
If inspection of a storage device is considered very likely, the steganographer may attempt to barrage a potential analyst with, effectively, misinformation. This may be a large set of files encoded with anything from random data, to white noise, to meaningless drivel, to deliberately misleading information. The encoding density on these files may be slightly higher than the "real" ones; likewise, the possible use of multiple algorithms of varying detectability should be considered. The steganalyst may be forced into checking these decoys first, potentially wasting significant time and computing resources. The downside to this technique is it makes it much more obvious that steganographic software was available, and was used.
Conclusions and further action
Obtaining a warrant or taking other action based solely on steganalytic evidence is a very dicey proposition unless a payload has been completely recovered and decrypted, because otherwise all the analyst has is a statistic indicating that a file may have been modified, and that modification may have been the result of steganographic encoding. Because this is likely to frequently be the case, steganalytic suspicions will often have to be backed up with other investigative techniques.
