Extensions to Higher Order Statistics

The proposition that a neuron might be in the business of finding the centres of clusters, in dynamical terms, has implications for what we have suggested is the mechanism for UpWriting, as well as general force in the context of unsupervised learning. This is rather different from the model of a neuron as binary classifier or adaptive threshold logic unit. It has far more similarity to the Kohonen network, but that is matter of which I shall not speak, as Gandalf puts it. But can this model be carried further? There are arguments for thinking it can.

A neuron in the visual cortex of a cat is presented with a flood of data concerning the edge orientations painted on the wall of its box: it can do temporal integration and the scene changes continuously over times shorter than the saccadic jumps of the eye. The inputs to a single neuron will almost invariably be correlated, and the neuron may be able to learn this fact. If the modification of synaptic transmitter substance or synaptic geometry, whatever records the effect of the datum on the neuron, is accomplished by a process which admits of interactions between synapses, then the correlations may also be learnt. To put it in geometric terms, the neuron sees not a single datum but a cluster of points in the input space, and the neurone may adapt its state to respond to their distribution. In simple terms, instead of responding to merely the location of the data points, it may be modelling higher order moments than the zeroth and first.

We know that the neuron has a response which is not infinitely sharp, and instead of representing it as a point or dog sitting in the space, trying to sit over a rabbit or cluster of rabbits, we might usefully think of it as conveying shape information. If for some reason we wanted to stop at seond order instead of first, we might visualise it as an ellipse, trying to do a local modelling, and the response to a subset of the data might be modelled by something not too unlike a gaussian distribution specified by the form represented by the ellipse. Thus a family of neurons may be dynamically achieving something rather like a gaussian mixture model of a data set. The complications avoided by stopping at second order make this quite pleasing, but of course there is no particular reason to think that real neurons do anything so simple In general we may however model them as computing moments up to some order less than n, for data in ${\fam11\tenbbb R}^n$, where n is of order the number of afferent synapses of the neuron. This may be typically around 50,000, according to current estimates.