MNIST database


The MNIST database is a large database of handwritten digits that is commonly used for training various image processing systems. The database is also widely used for training and testing in the field of machine learning. It was created by "re-mixing" the samples from NIST's original datasets. The creators felt that since NIST's training dataset was taken from American Census Bureau employees, while the testing dataset was taken from American high school students, it was not well-suited for machine learning experiments. Furthermore, the black and white images from NIST were normalized to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels.
The MNIST database contains 60,000 training images and 10,000 testing images. Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset.
The original creators of the database keep a list of some of the methods tested on it. In their original paper, they use a support-vector machine to get an error rate of 0.8%. An extended dataset similar to MNIST called EMNIST has been published in 2017, which contains 240,000 training images, and 40,000 testing images of handwritten digits and characters.

Dataset

The set of images in the MNIST database is a combination of two of NIST's databases: Special Database 1 and Special Database 3. Special Database 1 and Special Database 3 consist of digits written by high school students and employees of the United States Census Bureau, respectively.

Performance

Some researchers have achieved "near-human performance" on the MNIST database, using a committee of neural networks; in the same paper, the authors achieve performance double that of humans on other recognition tasks. The highest error rate listed on the original website of the database is 12 percent, which is achieved using a simple linear classifier with no preprocessing.
In 2004, a best-case error rate of 0.42 percent was achieved on the database by researchers using a new classifier called the LIRA, which is a neural classifier with three neuron layers based on Rosenblatt's perceptron principles.
Some researchers have tested artificial intelligence systems using the database put under random distortions. The systems in these cases are usually neural networks and the distortions used tend to be either affine distortions or elastic distortions. Sometimes, these systems can be very successful; one such system achieved an error rate on the database of 0.39 percent.
In 2011, an error rate of 0.27 percent, improving on the previous best result, was reported by researchers using a similar system of neural networks. In 2013, an approach based on regularization of neural networks using DropConnect has been claimed to achieve a 0.21 percent error rate. In 2016 the single convolutional neural network best performance was 0.31 percent error rate. As of August 2018, the best performance of a single convolutional neural network trained on MNIST training data using realtime data augmentation is 0.26 percent error rate. Also, the Parallel Computing Center obtained an ensemble of only 5 convolutional neural networks which performs on MNIST at 0.21 percent error rate. Some images in the testing dataset are barely readable and may prevent reaching test error rates of 0%. In 2018 researchers from Department of System and Information Engineering, University of Virginia announced 0.18% error with simultaneous stacked three kind of neural networks .

Classifiers

This is a table of some of the machine learning methods used on the database and their error rates, by type of classifier:
TypeClassifierDistortionPreprocessingError rate
Linear classifierPairwise linear classifierDeskewing7.6
Decision stream with Extremely randomized treesSingle model 2.7
K-Nearest NeighborsK-NN with non-linear deformation Shiftable edges0.52
Boosted StumpsProduct of stumps on Haar featuresHaar features0.87
Non-linear classifier40 PCA + quadratic classifier3.3
Random ForestSimple statistical pixel importance2.8
Support-vector machine Virtual SVM, deg-9 poly, 2-pixel jitteredDeskewing0.56
Deep neural network 2-layer 784-800-101.6
Deep neural network2-layer 784-800-10Elastic distortions0.7
Deep neural network6-layer 784-2500-2000-1500-1000-500-10Elastic distortions0.35
6-layer 784-40-80-500-1000-2000-100.31
Convolutional neural network6-layer 784-50-100-500-1000-10-10Expansion of the training data0.27
Convolutional neural networkCommittee of 35 CNNs, 1-20-P-40-P-150-10Elastic distortionsWidth normalizations0.23
Convolutional neural networkExpansion of the training data0.21
Random Multimodel Deep Learning 10 NN-10 RNN - 10 CNN0.18
Convolutional neural networkCommittee of 20 CNNS with Squeeze-and-Excitation NetworksData augmentation0.17