Newtonpedia ES
Home › Computing › Deep learning
Deep learning
Image: Wikimedia Commons
Computing · 1986

Deep learning

A machine learning technique that trains artificial neural networks with many successive layers to extract complex patterns from data.

Year
1986
People involved
Geoffrey Hinton, Yann LeCun
Section
Computing

History

Deep learning is a branch of machine learning that uses artificial neural networks built from many layers. Each layer turns the data it receives into a more abstract representation. In an image, for instance, early layers may pick out edges, later ones combine edges into shapes, and the final ones may recognise a face. What sets the approach apart is that the system works out for itself which features are worth extracting, rather than relying on rules written by experts.

Before this shift, much of machine learning depended on feature engineering, in which specialists decided which measurements of an image or signal would be fed to a classifier. Neural networks have an older story. In 1958, Frank Rosenblatt proposed the perceptron, a network with a hidden layer whose weights did not learn. Work on networks with more layers that could really learn stalled for years. Progress came in stages. In 1965, Alexey Ivakhnenko and Lapa published the group method of data handling, usually regarded as the first working deep learning algorithm, and a 1971 paper described an eight-layer network trained with it. In 1967, Shun'ichi Amari published a multilayer perceptron trained by stochastic gradient descent, a method that nudges the weights step by step to reduce the network's error.

A key piece was backpropagation, an efficient use of the chain rule from calculus that passes the output error backwards through the layers so each one can be corrected. A modern form appeared in Seppo Linnainmaa's 1970 master's thesis, and Paul Werbos applied it to neural networks in 1982. In 1986, David Rumelhart and his colleagues popularised it without citing the earlier work. That same year, Rina Dechter introduced the term "deep learning" to the machine learning community. Its origin is more tangled than a single date suggests, and the name took hold only gradually, so 1986 should be read as approximate.

Specific architectures followed. In 1979, Kunihiko Fukushima introduced the neocognitron, an ancestor of convolutional networks, though it was not trained by backpropagation. In 1989, Yann LeCun and his team built LeNet to read handwritten postal codes, and training took three days. LeNet-5, from 1998, was used by several banks to read handwritten amounts on digitised cheques. Meanwhile, in 1991, Jürgen Schmidhuber proposed a hierarchy of recurrent networks pre-trained one level at a time, and Sepp Hochreiter that same year identified the vanishing gradient problem, which makes long-range dependencies hard to learn. The answer was the long short-term memory network, published in 1995, and a forget gate added in 1999 became the standard form. Between 1985 and the mid-1990s, Geoffrey Hinton, Terry Sejnowski, Peter Dayan and others developed generative models such as the Boltzmann machine, which were more expensive to train than backpropagation. In speech recognition, neural methods took a long time to beat established statistical systems.

Reception was uneven. Interest faded in the 1990s and early 2000s, partly because labelled data and computing power were scarce. The turn came when graphics processing units, which run many calculations in parallel, began to be used to train networks. An OpenAI analysis noted that the computation used in the largest projects grew 300,000-fold between 2012 and 2017, doubling roughly every 3.4 months.

Today deep learning underpins computer vision, speech recognition and synthesis, machine translation and large language models built on transformers. It is also used in bioinformatics, drug design and medical imaging, where Vision Transformers have in some cases matched or exceeded convolutional networks. Although these systems take loose inspiration from the brain, they do not aim to reproduce its workings, and their success depends as much on data and hardware as on the algorithms themselves.

Timeline

  • 1965Ivakhnenko and Lapa publish the group method of data handling, regarded as the first working deep learning algorithm.
  • 1967Shun'ichi Amari publishes a multilayer perceptron trained by stochastic gradient descent.
  • 1979Fukushima introduces the neocognitron, an ancestor of convolutional networks.
  • 1986Rumelhart and colleagues popularise backpropagation, and Rina Dechter introduces the term deep learning.
  • 1989LeCun and his team create LeNet to read handwritten postal codes.
  • 1995The long short-term memory network is published, a recurrent architecture for long-range dependencies.
  • 1998LeNet-5 is used by banks to read handwritten amounts on cheques.
  • 2014Schmidhuber's adversarial principle is applied in generative adversarial networks (GANs).

Trivia and anecdotes

A name with a history

The year 1986 is linked to the term "deep learning", but its origin is tangled and the name took hold slowly. The exact date is therefore approximate.

The P in ChatGPT

According to Schmidhuber's work, the P in ChatGPT refers to pre-training, an idea he had explored in the 1990s.

People involved

Other milestones in Computing