1. DenoisingAutoencoder
Before the neural network model training phase begins, pre-training the model using an autoencoder can determine the initial parameter values of the encoder W. However, due to factors such as model complexity, the amount of training data, and data noise, the initial model obtained through an autoencoder often carries the risk of overfitting. For an introduction to autoencoders, please refer to: Autoencoder.
Before introducing the Denoising Autoencoder , let's review the overfitting problem in machine learning, as shown in the figure below, which illustrates the impact of the amount of training data and noise on model overfitting. The red area represents overfitting, the blue area represents underfitting, and the green area represents the desired model. As can be seen from the figure, the less training data and the more noise in the data, the more prone the model is to overfitting.
Given a fixed model complexity and data volume, one way to prevent overfitting is to reduce the amount of noise in the data, i.e., to clean the training set. However, what if we cannot detect and remove noise from the data? Another way to prevent overfitting is to add noise to the data. This seems to contradict the previous conclusion, but it is an effective way to enhance the robustness of the model. Taking handwritten digit recognition as an example, the Autoencoder first encodes the input image, then decodes and reconstructs the digit information in the original image after passing through the hidden layer. Suppose we now input an image with some noise, such as blemishes or tilted digits, and we still want the decoded image to be clean and correct. This requires the encoder to not only have encoding capabilities but also noise reduction capabilities. A model trained in this way has stronger robustness.
DenoisingAutoencoder builds upon Autoencoder by adding noise to the input data (the network's input layer) to prevent overfitting. This makes the learned encoder W more robust, thus enhancing the model's generalization ability. DenoisingAutoencoder was proposed by Bengio in 2008; for details, please refer to his paper.
Extracting and composing robust features with denoising autoencoders.
The diagram of DenoisingAutoencoder in the paper is shown below, where x is the original input data. DenoisingAutoencoder sets the values of input layer nodes to 0 with a certain probability, thus obtaining a noisy model input xˆ. This is very similar to dropout, except that dropout sets the neurons in the hidden layers to 0.
Bengio offered the following intuitive explanation of DenoisingAutoencoder:
DenoisingAutoencoder is similar to human perception. For example, when a human eye looks at an object, if a small part of the object is obscured, the human can still recognize it.
When people receive multimodal information (such as sound, images, etc.), the absence of some modal information sometimes does not have a significant impact.
The essence of an autoencoder is to learn an equality function, that is, the network input and the reconstructed output are equal. This equality function representation has a drawback: when the test samples and training samples do not conform to the same distribution, that is, when they differ greatly, the performance is poor. DenoisingAutoencoder has made progress in this regard.
Experimental code
For an explanation of the MNIST dataset format and data preprocessing code input_data.py, please refer to: Tutorial(2).
The experimental code is as follows:
