Newtonpedia ES
Home › Computing › Backpropagation
Computing · 1970

Backpropagation

An algorithm that computes how each weight in a neural network should change to reduce its error, by passing the error backward from output to input.

Year
1970
People involved
Seppo Linnainmaa
Section
Computing

History

Backpropagation is the procedure that makes it practical to train neural networks with several layers. Its core idea is simple: the network's output is compared with the correct answer, an error is measured, and that error is passed backward, layer by layer, to determine how much each connection contributed to the mistake. Each weight is then adjusted in the direction that reduces the error. As a result, the hidden layers gradually learn to detect useful features in the data without anyone programming them by hand.

Before this method, single-layer perceptrons were quite limited. They could separate data only with one simple boundary, and it was known that networks with several layers could handle more complex problems, but there was no practical way to adjust their weights. The difficulty was determining how much of the error belonged to each hidden neuron, since those layers had no direct target values.

The modern form is most closely associated with Seppo Linnainmaa, who in 1970, in his master's thesis at the University of Helsinki, described a method for computing derivatives of complex functions by accumulating the calculations in reverse. This technique is mathematically the core of backpropagation, although Linnainmaa did not apply it to neural networks at that time. Other contributions were made independently by several researchers in the following decades, so the history of the method has several partial authorships and a terminology that is not always consistent.

The application to training neural networks gained wider traction in the 1980s. In 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams published a paper that showed clearly how multilayer networks could be trained with this procedure, and that publication helped popularize it in the scientific community. Since then the method has been used with ever larger networks, although for a time its practical effectiveness depended on the computing power available.

Underneath, the method relies on gradients, that is, on measuring how the error changes when a weight is slightly modified. The aim is to move in the direction that lowers the error, step by step. Backpropagation avoids repeated work by reusing results from later layers, which makes it far faster than computing each derivative separately.

Today it is the mechanism behind the training of most deep learning systems, from speech and image recognition to machine translation and language models. Although more sophisticated optimizers and variants have been developed for specific problems, nearly all of them rest on the same principle of propagating the error backward to adjust the network's parameters.

Timeline

  • 1970Seppo Linnainmaa describes in his master's thesis the reverse accumulation method, the mathematical basis of backpropagation.
  • 1986Rumelhart, Hinton and Williams publish a paper showing how to train multilayer networks with this procedure.

Other milestones in Computing