Boltzmann Machines

The Man Who Believed in Brains When Nobody Else Did

Geoffrey Hinton comes from extraordinary stock. His family tree includes George Boole, whose algebra of logic became the foundation of modern computing, and George Everest, after whom the mountain is named. There was apparently enormous pressure to do something significant. He has. His Nobel citation, shared with John Hopfield in 2024, awards them jointly "for foundational discoveries and inventions that enable machine learning with artificial neural networks." In plain terms: they built the intellectual foundations on which today's AI runs.

But Hinton's path was never straightforward. From the early 1970s onward, he believed that the right way to build intelligent machines was to take inspiration from the brain — to use networks of simple interconnected units, like neurons, that could learn patterns from data. This was a minority view, even a derided one. The mainstream of AI research believed in logic, in hand-crafted rules, in symbols. Neural networks, the establishment felt, were a dead end. Hinton disagreed. He kept working. For decades, his funding was thin, his audience small, and his ideas considered quaint.

He moved from Britain to the United States, then to Canada — partly, he has said, out of discomfort with American military funding of AI research. He set up at the University of Toronto, where he trained a generation of researchers who would go on to reshape the entire field. His lab became, quietly, the most consequential workshop in modern computing.

Energy, Memory, and the Physics of Thought

To understand what Hinton built, you first need to understand what his co-laureate John Hopfield contributed in 1982. Hopfield drew an analogy between magnetic materials in physics and networks of artificial neurons — and the analogy turned out to be surprisingly deep. This is why the Nobel Prize is in Physics: at its root, this work borrows the mathematical language of statistical mechanics, the physics of systems with enormous numbers of interacting parts.

A Hopfield network is a collection of binary neurons — each one either on or off. Every pair of neurons is connected by a weighted link, which can be positive (encouraging both to fire together) or negative (discouraging it). The key insight is that every possible configuration of the network — every pattern of ons and offs — has an associated energy. Some configurations have low energy; some have high energy. And the network, left to its own devices, naturally rolls downhill toward lower energy, like a ball finding the bottom of a valley.

This was a beautiful and genuinely new idea. But it had two serious limitations. First, the network could only store a limited number of memories before they interfered with each other. Second — and more fundamentally — Hopfield nets always rolled straight to the nearest valley. If that valley wasn't the best one, there was no way out. They got trapped.

Adding Noise: How to Escape a Bad Valley

Hinton's key move, developed with Terry Sejnowski in 1983, was to introduce randomness. Instead of neurons that always made the locally optimal decision, what if neurons occasionally made random, even uphill, decisions? A neuron receiving strong input would still usually fire — but occasionally it wouldn't. A neuron receiving weak input would occasionally fire anyway. The probability of each decision depended on the temperature of the system — a concept borrowed directly from statistical physics.

This is the Boltzmann machine. At high temperature, the system is highly random — neurons behave almost unpredictably, and the network wanders freely across its landscape. As the temperature drops, the randomness decreases, and the network settles — but because it explored freely at high temperature, it has a much better chance of landing in a genuinely good valley rather than just the nearest one. This process is called simulated annealing, named after the metallurgical technique of slowly cooling metal to remove defects.

At thermal equilibrium — the state the network reaches after enough random updates — the probability of any configuration is determined entirely by its energy. Low-energy configurations are more probable. High-energy ones are rare. This is the Boltzmann distribution from physics, and it is what gives the machine its name.

Wake, Dream, Repeat: How the Machine Learns

A network that settles to good configurations is interesting, but the deeper question is: how does it learn? How do you shape the energy landscape so that the valleys correspond to things you actually want — real faces, real words, real patterns in data?

Hinton and Sejnowski's answer was an elegantly simple two-phase rule. In the wake phase, the network is shown real data — an image is clamped onto the visible neurons. The hidden neurons settle into their interpretation of that image. Whenever two connected neurons are both active at the same time, their connection weight is nudged upward. The network is, in effect, learning to associate what it sees with the interpretations it forms.

In the sleep phase, the data is removed and the network is left to dream — to generate its own activity freely. Again, whenever two connected neurons fire together, their connection weight is nudged — but this time downward. The network is unlearning its fantasies, reducing the energy of patterns it generates in its sleep.

What makes this rule remarkable is its locality. Each connection only needs to know two things: how often its two neurons fire together during waking, and how often they fire together during sleep. It doesn't need a central controller sending error signals back through the whole network. Every synapse learns from purely local information — which is also, strikingly, how synapses in the biological brain are thought to learn.

The Shortcut That Changed Everything

The original Boltzmann machine was beautiful but brutally slow. Reaching thermal equilibrium required many thousands of random updates; learning from a large dataset was practically infeasible. For nearly two decades, the idea sat on a shelf — theoretically elegant, practically unusable.

Then, in the early 2000s, Hinton found a shortcut. By restricting the architecture — removing connections between hidden neurons so that only visible-to-hidden connections remained — the wake phase collapsed to a single parallel update. The network reached equilibrium in one step rather than thousands. These simplified networks, called Restricted Boltzmann Machines, could be trained rapidly. And crucially, they could be stacked: train one layer of features, then treat those features as the data for a second layer, and so on, building a hierarchy of increasingly abstract representations.

In 2006, Hinton and Ruslan Salakhutdinov published a paper showing that a stack of RBMs could be used to initialise a deep neural network — one with many layers — in a way that made backpropagation work far better than anyone had managed before. This was the spark. Deep learning, which had struggled for decades against the practical problem of training many-layered networks, suddenly had a working method. Within six years, Hinton's student Alex Krizhevsky, working with Ilya Sutskever, built AlexNet — a deep convolutional network that entered the 2012 ImageNet image recognition competition and beat every other entry by nearly eleven percentage points. It was, in the words of researcher Yann LeCun, "an unequivocal turning point in the history of computer vision."

Hinton himself remarked, with characteristic self-deprecating precision, that with AlexNet: "Ilya thought we should do it, Alex made it work, and I got the Nobel Prize."

The Man Who Lit the Fire, and What He Thinks of the Flames

In May 2023, Geoffrey Hinton resigned from Google, where he had worked for a decade after the AlexNet acquisition. He wanted, he said, to speak freely. What he wanted to say was this: the technology he had spent his life building might be one of the most dangerous things humanity has ever created.

He is careful not to be fatalistic. AI, he says, will do enormous good — in medicine, in drug discovery, in understanding the physical world. He has no regrets about the work itself. But he estimates a ten to twenty percent chance that AI systems, within the next few decades, will become smarter than humans in general terms — and that this transition, if it happens without sufficient care, could go very badly. He draws the comparison to Robert Oppenheimer, who built the atomic bomb, campaigned against the hydrogen bomb, and found the world beyond his control. He reminds us that Oppenheimer's warning was not heeded fast enough.

There is a particular irony in reading this lecture alongside the technology you are likely using right now. The distillation you're reading was assisted by a large language model — one of the direct descendants of the deep learning revolution that Hinton's stacked RBMs helped ignite. The very tool used to make his ideas accessible is the fruit of those ideas. Hinton is aware of this loop. He is not asking us to stop. He is asking us to think — carefully, urgently, with more seriousness than the pace of commercial AI development currently allows.

The lecture itself is spare: a sequence of technical slides, terse and precise. But behind it is fifty years of unfashionable belief, two decades of patient waiting for the world to catch up, and a conclusion that arrives with the weight of a man who knows exactly what he set in motion. It is worth reading slowly, and sitting with.