Newtonpedia ES
Home › Computing › Hopfield network
Computing · 1982

Hopfield network

A recurrent neural network with symmetric connections that works as an associative memory: it recovers a complete pattern from a partial or distorted version.

Year
1982
Section
Computing

History

A Hopfield network is a type of artificial neural network in which every unit, or neuron, connects to all the others except itself, and the connections are symmetric. Its main interest lies in acting as a content-addressable memory: instead of looking up the place where a piece of data is stored, the network rebuilds the full memory from a partial or noisy cue, much as we recognise a song from just a few notes.

Similar ideas existed before. Human cognitive psychology had long studied associative memory, the kind that calls up one memory from another that resembles it. Frank Rosenblatt studied in the 1960s perceptron networks with recurrent connections, and W. K. Taylor proposed in 1956 a model trained with Hebbian learning, the rule by which connections between units that are active together grow stronger. Karl Steinbuch published the Lernmatrix in 1961, and Teuvo Kohonen trained an associative memory by gradient descent in 1974. From statistical physics came another ingredient. The Ising model, published in the 1920s to describe magnetism, defines an energy that measures how stable a configuration is. Roy Glauber studied in 1963 how this model evolves over time, and several researchers, including Shun'ichi Amari in 1972, proposed using Hebbian learning to store patterns in an Ising-type model.

John Hopfield, a American physicist, published in 1982 a paper showing that a network of binary units, which take only two values such as 1 and -1, can store several patterns. Each pattern is written by adjusting the connection weights according to Hebb's rule. Afterwards the network updates its units one at a time, and its energy decreases until it settles into a local minimum that matches a stored pattern. Energy here is a quantity the network tends to lower at each step, rather like a ball rolling into the bottom of a valley. In 1984 he extended the model to continuous values, and the work became a standard model for studying neural networks from the viewpoint of physics.

The network has clear limits. It does not guarantee reaching the correct pattern, since it sometimes falls into a false stable state, and its storage capacity is small compared with its number of neurons. Its binary neurons and its incompatibility with gradient-based training mean that classical versions are rarely used in modern machine learning. Even so, in 2016 Dmitry Krotov and Hopfield proposed a variant with much greater capacity, known as dense associative memory or modern Hopfield network, which revived interest in the model. In 2024 Hopfield and Geoffrey Hinton received the Nobel Prize in Physics for their foundational contributions to machine learning, including this network.

Today Hopfield's ideas form part of the theoretical background of artificial intelligence. They link statistical physics with the study of the brain and with methods for retrieving information, and his view of memory as reconstruction from cues still shapes modern systems.

Timeline

  • 1956W. K. Taylor proposes an associative memory model trained with Hebbian learning.
  • 1963Roy Glauber studies the time evolution of the Ising model, a key step towards recurrent networks.
  • 1974Teuvo Kohonen trains an associative memory using gradient descent.
  • 1982John Hopfield publishes the paper describing the network with binary units and its energy function.
  • 1984Hopfield extends the model to neurons with continuous values.
  • 2016Krotov and Hopfield propose a version with large storage capacity, now called the modern Hopfield network.
  • 2024Hopfield and Geoffrey Hinton receive the Nobel Prize in Physics.

Trivia and anecdotes

A model borrowed from magnetism

The network takes its energy function from the Ising model, developed to describe magnetic materials. This link between physics and neuroscience gave a new way of thinking about memory.

A Nobel Prize in 2024

In 2024, Hopfield and Geoffrey Hinton shared the Nobel Prize in Physics for their contributions to machine learning.

Other milestones in Computing