Theory of the Network

The theory of the network is based on statistical mechanical concepts. In particular, it can be shown that when the network is allowed to run according to the simulated annealing procedure, the states of the network occur with a fequency that depends on the Boltzmann distribution. This distribution is a fundamental one in statistical mechanics. The distribution is determined by a single parameter, which is the temperature in thermodynamical situations, and has the property that states with low energy are unlikely to occur when the temperature parameter is large, and much more likely to occur when the temperature parameter is small.

The Boltzmann distribution provides the rationale for the simulated annealing procedure. Starting the procedure at high temperature makes the network more likely to occupy high energy states. As the temperature is reduced, the low energy states become more likely. Very low temperatures favour very low energy states, which should be close to the global minimum.

The Boltzmann distribution also provides the rationale for the procedure used to update the weights. It can be shown that the quantity $p_{ij} - p^\prime_{ij}$, described above, is an indicator of the direction and amount of the change in w_ij that will result in a better matching of the input and output patterns.