The death of Neural Nets

The late sixties saw the publication of Minsky and Papert's Perceptrons. This casually identified all perceptrons in the sense of Rosenblatt with the elementary alpha-perceptron, preceded by some local process which `extracted features' from an image by looking at only some of the pixels within a region, and returned a vector of binary results which was then the input to a single model neuron. This did less than justice to the possibilities for (a) looking at the whole image and (b) using many units and hence having piecewise affine functions instead of just affine functions. By pointing out the limitations of this rather skeletal abstraction in a, perhaps, rather confusing manner, Minsky and Papert managed to convince most readers that there were easy problems which their brains could solve which the perceptron model family could not. The cover of the book showed two diagrams, one a double spiral and the other a single spiral, and explained that it was hard for a perceptron to tell the difference. It was pretty hard for the reader, too, and involved some tracing of paths with fingers, but this was overlooked. Similarly, it was shown that combining local viewpoints could never guarantee to tell if a pixel set is connected or not. Rather like the proof that no (finite state) computer can add any two numbers, which it somewhat resembles, this is a result of limited practical utility. The whole range of all possible perceptrons, in the sense of Rosenblatt, would encompass what are now called recurrent nets as well as time delay nets and indeed just about all the extant neural nets. Minsky clobbered only a limited class, but he used Mathematics, and so hardly anybody troubled to read the fine print. What he was in fact trying to tackle was the issue of when local information about the geometry of an object could be condensed by a linear or affine map and then combined with other such local compressions to give global properties of the object. There is some reason to doubt that this is a good problem. There is even more reason to doubt if this problem has a lot to do with how brains work. Still, it is no dafter than much Mathematical research and was harmless in all respects except its effect on Neural Net work.

The moral to be drawn from this would seem to be that most people don't read books very carefully, especially if they contain mathematics. It might have helped if Minsky hadn't had such a high reputation, or if he had taken more care to explain that the limitations of the systems of the sort he was analysing were not pertinent to the entire model class. Minsky took it that the input to a perceptron was a discrete retina always, and that the values were binary only, and that it was impractical to look at other than local regions or to use other than affine functions. Thus the limitations of the perceptrons Minsky analysed were fairly severe from the beginning, and it was unfortunate that sloppy reading left the many with a vague notion that all perceptrons were being disposed of. There is a suspicion that the critical tone of the book was the result of a disenchanted love affair: the AI community had imagined that all their problems would be solved by just bunging them into a huge neural net and waiting for the computer to grind to a conclusion. Ah well, the sixties were like that. If MIT had Perceptrons, the hippies had flower power. Both sobered up eventually. MIT AI Lab did it first, but at the expense, perhaps, of a bad case of the snits.

The belief that Minsky had shown that Perceptrons couldn't do anything non-trivial, came at about the same time that disenchantment was beginning to set in among funding officers. It is folk-lore that Rosenblatt had spent around Two Million US Dollars by the time of Minsky's book and had little to show for it, at least by the standards of the military. So it is possible that Minsky merely drove the nail a little further into the coffin. Anyway, something around this time more or less stopped research on piecewise affine systems dead in its tracks. But you can't keep a good idea down, and eventually by changing

the name to Artificial Neural Nets (ANNs) and later MultiLayer Perceptrons (MLPs) (when the variety of these fauna started to proliferate), they came back. They are now the staple of snake oil merchants everyhere, alas, and we need another Marvin Minsky to come again and tell us what they can't do. But this time, do it right.

For Minsky's comments on the dispute, see Whence Cybernetics, Connections, The Newsletter of the IEEE Neural Networks Council, Vol.3. No.3., p3., September 1993. And also P4. It should be remarked that some of the views expressed (by the original disputants, not by Minsky) show an extensive innocence.