Network Structure

The Neocognitron is designed as a system to recognize the handwritten digits from 0 to 9. It has a hierarchical structure, being divided into nine layers.

The first layer is the input layer and is designated U₀. It consists of an array of photoreceptor units arranged in a $19 \times 19$ square. The remaining layers are divided into four sets of two layers each. Each of these sets has a U_S layer and a U_C layer, which are given numbers to indicate the set to which they belong. The flow of information is along the path $U_0 \rightarrow U_{S1} \rightarrow U_{C1} \rightarrow U_{S2} \rightarrow U_... ... \rightarrow U_{S3} \rightarrow U_{C3} \rightarrow U_{S4} \rightarrow U_{C4}$.

The U_S1 layer has $19 \times 19 \times 12$ units. Each unit in the U₀ layer has 12 units corresponding to it in the U_S1 layer. Each of these units receives inputs from the corresponding units in the U₀ layer and from the eight units surrounding it. Each of these twelve units is trained to respond to a pattern that represents a small portion of a line segment. In this way, the S1 units function as a collection of line segment detectors spread over the entire input array.

The U_C1 layer consists of $11 \times 11 \times 8$ units. The 12 bit patterns recognized by the S1 units actually correspond to 8 possible orientations of line segments, so outputs from equivalent S1 units are fed to the same C1 units. Each C1 unit receives inputs from S1 units corresponding to a $5 \times 5$ square of receptor units. This reduces the dimensions of the layers from $19 \times 19 \times 12$ to $11 \times 11 \times 8$.

There are similar arrangements between the succeeding layers, so that the U_S2 layer has $11 \times 11 \times 38$ units, the U_C2 layer has $7 \times 7 \times 22$ units, the U_S3 layer has $7 \times 7 \times 32$,the U_C3 layer has $7 \times 7 \times 30$, the U_S4 layer has $3 \times 3 \times 16$, and the U_C4 layer has ten units, which are used to indicate which of the digits has been presented to the network. The gradual reduction in the first two dimensions of each layer reflects the gradual increase in the proportion of the input array which affects each unit. The alternating reduction and increase in the size of the third dimension reflects encoding of a number of features in the output from the previous layer and the subsequent reduction of this number by identifying features that are equivalent for the purposes of digit recognition.

The units described above are all excitatory units. In addition, there are cells between each of the U_C layers and the following U_S layer which have an inhibitory effect on the units in the following layer. A little arithmetic shows that there are more than 50000 units in the Neocognitron.

The Neocognitron is a feedforward network with the outputs from each layer (except the last) becoming the inputs to the next layer. It is not a fully connected network; there are no connections between units belonging to the same layer, and each unit is connected to some, rather than all, of the units in the previous layer. This means that the number of connections is less than that of a Backpropagation network of similar size and far less than that of a comparable Hopfield network. Even so, the Neocognitron has about 14 million connections.