Committees vs Back-Propagation

It is natural to wonder whether there are any advantages in using Back-propagation over a committee. The committee is experimentally found to be faster than a three layer net by about an order of magnitude and either converges fairly quickly or never converges because the problem is unsolvable by the given committee. The three layer back-propagation net sometimes converges and sometimes doesn't, depending on the initial state. There is a theorem which justifies the back-propagation algorithm, sort of, but no actual proof of convergence. The state space is bigger for back-propagation, which gives more flexibility, but flexibility is a property of jellyfish and earthworms which hasn't done them a lot of good when it comes to arguing the toss with insects and vertebrates. There are more of us than of them. So one should not be unduly influenced by such simple arguments as the unbridled advantages of the larger state space.

Experiments suggest that the so called three layer net has no practical advantages over a committee of slightly larger size and an equal number of parameters, and indeed is likely to take longer to converge on average. It is fairly easy to understand the thinking behind a committee net, while back-propagation tends to numb the mind of the average user for whom calculus of more than one variable has essentially the status of black magic in the wilder parts of New Guinea. Unfortunately, superstition is rife these days, and the incomprehensible is thought to have manna not possessed by the intelligible.

For four layer neural nets, the articulated four layer net with a fixed number of units in the second layer partitioned into groups intended to cooperate with each other in surrounding a convex region of points of some type, the partitioned groups then being ORed with each other to give a decomposition of the data as a union of convex regions, again outperforms the four layer back-propagation experimentally, being about an order of magnitude faster on sets of moderate complexity with greater relative improvement on more complex sets. Mind you, this is a competition between losers, there are better ways. It is true that sometimes back-propagation can eventually find a solution which the articulated net cannot, but at least with the articulated net you find out that there is no solution with your given geometry rather faster.