SANN Overviews - Pre and Post Processing of Data
All neurons in a neural network take numeric input and produce numeric output. The activation function of a neural unit can accept input values in any range and produces output in a strictly limited range. Although the input can be in any range, there is a saturation effect so that the unit is only sensitive to inputs within a fairly limited range. For example consider the logistic function. In this case, the output is in the range (0,1), and the input is sensitive in a range not much larger than (-1,+1). Thus, for a wide range of input values ranging outside (-1, +1), the output of a logistic neuron is approximately the same. This saturation effect will severely limit the ability of a network from capturing the underlying input-target relationship.
The above problem can be solved by limiting the numerical range of the original input and target variables. This process is known as scaling, which is one of the most commonly used forms of preprocessing. STATISTICA Automated Neural Networks scales the input and target variables using linear transformations such that the original minimum and maximum of every variable is mapped to the range (0, 1).
There are other important reasons for standardization of the variables. One is related to weight decay. Standardizing the inputs and targets will usually make the weight decay regularization more effective. Other reasons include the original variable scaling and units of measure. It is often the case that variables in the original data set have substantially different ranges (i.e., different variances). This may have to do with the units of measurements or simply the nature of the variables themselves. However, the numeric range of a variable, among many, may not be a good indication of how important that variable is.