In the `classical' theory of Neural Nets, which we may trace to Rosenblatt, Block, Widrow and Nilsson, not forgetting the early work of Pitts and McCulloch, there are two strands. One is the idea of piecewise affine (or piecewise linear as they used to say in the bad old days) hypersurfaces as discriminating boundaries, and the other is the belief that this is how brains might do the job of classifying data. I have already discussed the former in the case of the archetype pattern recognition problem of telling the guys from the gals.
The idea that a neuron might be a threshold logic unit, that is to say a unit where some weighted sum of inputs was compared with a threshold and either output a 1 or -1, according to whether the weighted sum exceeded the threshold or not, goes back to the observation that sometimes neurons fired, in which case a spike of voltage difference between inside and outside the neuron travelled along it, from input end to output, afferent dendrites to efferent dendrites, until it came to another neuron, at which point the following neuron, given a sufficient amount of stimulus from the first and possibly other neurons, might also fire. So the firing of a neuron came to be seen as an all or none affair, mediated by the transmission coefficients at the synapses, the places where one neuron impinged on another. This made it a weighted majority gate, in an alternative terminology, with each weight representing some state of a synapse. The Hebb doctrine, that memory was stored in transmitter substance density and possibly neuronal geometry, also contributed to the view of neurons as adaptive threshold logic units. This is not a view which has survived, but it had a substantial effect on the thrust of the development of algorithms in the classical period from about 1940 to the sixties.
The sketch of a `typical' neuron, as in Fig.5.1 is a biological version of a big brother of Fig.1.6, with more inputs. Engineers tend to prefer straight lines and right angles to the wiggly curves of biology.
Rosenblatt, in 1961, had seen the elementary perceptron, just a single one of these units, as a model for a single neuron, and conceded without shame that the critical issue of how more units could combine to give larger perceptrons was still open. He used a particular adaptation rule called the
alpha rule for training his elementary perceptron to output either +1 or -1, for use as a Pattern Classifier. It was well known that if the inputs were taken from the so called XOR data set, which had points at (0,0), (0,1), (1,0) and (1,1) where the first and last were of category 0 and the middle two of category 1 (and the name being chosen for reasons obvious to anyone with any familiarity with logic) then the elementary alpha perceptron could not classify the data correctly. This is rather simple geometry. The elementary perceptron with input the coordinates of points in the plane was simply an oriented line which fired if the input data was on one side and didn't if it was on the other. And the XOR data set was not linearly separable.
 |
Nilsson in 1965 had shown how what he called a `committee net' of three neurons, all fed the same input, could all feed output to a final neuron, the final neuron seeing the preceding layer as voting on the correct classification of the datum. By simply taking the majority verdict, easily accomplished by a suitable choice of weights, the final neuron could yield the correct classification. We shall return to this shortly.
