2. Hopfield network
Memory as an energy minimum.
A Hopfield network stores patterns in its couplings. Start from a corrupted pattern and apply deterministic updates to descend toward an attractor.
Current state
Network state
Start from a stored pattern or corrupt it.
Stored target: X
Hamming distance: 0 of 9 spins.
Energy: 0.000
Energy descent
Each update cannot increase the energy
How a Hopfield network learns
Hebbian learning reshapes the energy landscape.
A Hopfield network does not receive its weights by hand. For every training pattern ximu, it strengthens pairs of spins that agree, weakens pairs that disagree, and learns a one-spin bias:
This four-spin example learns two training data: A = ++-- and B = +-+-. Move through the data stream and inspect the couplings and every possible state energy.
Learned parameters
Weight matrix W and bias h
Diagonal entries are fixed to zero.
Exact energy landscape
All 16 configurations
Lower bars are lower-energy states.
The energy is E(s) = -1/2 sum(i,j) Wijsisj - sum(i) hisi. The learned bias h breaks the global spin-flip symmetry, so A and B are lower than -A and -B. In the special zero-bias Hopfield model, those pairs would be degenerate.