Whether it’s completing sentences, predicting protein structures or uncovering hidden patterns in data – today’s AI systems sit on foundations laid by work recognised by the 2024 Nobel Prize in Physics. Learn more about the origins of the revolutionary field of machine learning, that was once dismissed as “uninteresting”.
What if someone asked you to name the American sitcom about young people sharing an apartment in New York?
Your brain would probably serve you up the answer Friends followed by a wave of thoughts and images: the cafe ‘Central Perk,’ the characters Phoebe, Joey, Ross, Monica, Chandler and Rachel. You might even start to hum the theme song.
What your brain just performed is called associative memory. It is one of the many puzzles surrounding intelligence, artificial or otherwise. But how does your brain perform this trick?
Connections between neurons
At the time, John Hopfield was a young physicist. Understanding neurons and the human brain were not on his mind particularly – but would later become key for his career.
Hopfield worked at Bell Labs alongside Philip Anderson who would be awarded the Nobel Prize in Physics for work on the electronic structure of magnetic and disordered systems.
“Anderson was the best condensed-matter theoretical physicist alive. I was a beginner. No physicist knew about AI at that time.”
John Hopfield
Anderson was particularly interested in special types of random and disordered magnetic materials called spin glasses. “Phil was interested in phase transitions; he believed that spin glasses would have fascinating and important properties,” adds Hopfield. “I listened to Phil.”
The magnetic moments, or spins, of the atoms in spin glasses point in random directions, interfering with neighbouring atomic magnets. Stability is deeply challenging and the mathematics of this instability fascinated Anderson and Hopfield, as well as many other physicists, who during the 1970s searched for a way to describe the mysterious and frustrating spin glasses.
Hopfield appreciated the complexity and importance of spin glass physics as his curiosity took him beyond physics to start working on biological questions. In 1980, Hopfield accepted a professorship in chemistry and biology at Caltech, Southern California. Somewhat unexpectedly, the physics and mathematics Hopfield learnt from studying spin glasses would prove crucial when he started work on a new problem at Caltech: memory recall and associative memory. Conventional computer memory retrieves information by using an index to locate it in a fixed place. Thinking back to spin glass physics, Hopfield had a groundbreaking idea about how memories could be stored and retrieved in a more distributed system: a neural network.
He recalls that, at that time, no one working in the field of AI saw any connection to spin glasses. Neuroscientists often lacked training in the mathematics physicists learned, a significant blind spot that meant they were missing important neuroscience applications that could migrate from physics.
A neural network
Using the mathematics and physics of interacting spins Hopfield created a simple neural network model. This so-called Hopfield network could remember a pattern – like the way soft clay remembers a hand when pressed into it. But more than that, the network could retrieve the full handprint reasonably accurately from just a few details: some fingertips or a palm.
Another remarkable property of Hopfield networks is that they can store several patterns at the same time.
Hopfield compared retrieving a pattern from the network to rolling a ball through a landscape of peaks and valleys, with friction gradually slowing its movement. Drop the ball in a particular place and it will roll downhill to stop in a valley floor. In the same way, when the network is given an incomplete or damaged version of a stored pattern, its neurons repeatedly update until the network settles into a low-energy state: usually the remembered pattern associated with that region of the landscape.
Connections between artificial neurons have different weights or strengths and these encode relationships between stored patterns. This shapes the valleys in the landscape. This allows the network to reconstruct a complete memory from a fragment – rather like recognising the cast of Friends from only part of an image. Hopfield had created an associative memory: a system that could retrieve something from partial information.
Detect patterns in data
At one of Hopfield’s lectures on neural networks, a young psychologist named Geoffrey Hinton was in attendance. Hinton was intrigued by the Hopfield network and saw how it could be extended using ideas from statistical physics. Hinton was also inspired by the nineteenth-century physicist Ludwig Boltzmann, whose statistical mechanics describes how likely a physical system is to occupy different energy states at a given temperature.
Working with Terrence Sejnowski and others, Hinton developed a new kind of neural network called the Boltzmann machine. Unlike the Hopfield network, which retrieves particular stored patterns, a Boltzmann machine learns the probability distribution underlying a whole set of examples. Its connections are adjusted so that patterns resembling the training data become more probable. Once trained, it can recognise familiar features, fill in missing information and generate new examples of the kind of pattern on which it was trained – making it an early generative model.
Hopfield had found a way for a network to reconstruct a stored pattern from fragments; Hinton found a way for a network to learn what a family of patterns looks like and generate new members of that family.
In October 2024, the Royal Swedish Academy of Sciences decided to award the Nobel Prize in Physics to John Hopfield and Geoffrey Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.” The committee acknowledged Hopfield for his network and Hinton for his Boltzmann machine.
“The laureates’ work has already been of the greatest benefit. In physics we use artificial neural networks in a vast range of areas, such as developing new materials with specific properties.”
Ellen Moons, Chair of the Nobel Committee for Physics
Since the early work in the 1980s, it has taken decades and many twists and turns before this knowledge fully emerged into the public consciousness. However, anyone who has used a large language model can glimpse its origins in these early ideas. The energy landscapes of spin glasses, the distributed memories of Hopfield networks and the probabilistic modelling of Boltzmann machines were important forerunners of today’s AI systems that write fluently, predict protein structures and uncover hidden patterns in data. This new world has opened up as a result of a scientific culture where ideas from many disciplines can collide, and a wider culture that values curiosity.
