Companion to the 15-minute lecture

From deterministic memory to probabilistic learning.

These three small models follow the lecture sequence. Each calculation is exact, so every number can be inspected rather than taken on trust.

1. Hopfield to Boltzmann

Replace a threshold by a probability.

For one binary neuron, its local field h is the weighted input plus bias. A Hopfield net uses its sign. A Boltzmann machine samples from the same local energy difference at finite temperature.

Hopfield: s = 1 when h > 0   |   Boltzmann machine: p(s = 1) = 1 / [1 + exp(-h/T)]

Hopfield, T = 0

s = 1deterministic threshold

Boltzmann machine

p(s = 1) = 0.73last sample: s = 1

Sigmoid activation at the selected temperature

2. Hopfield associative memory

A learned energy minimum can repair a corrupted pattern.

This three-unit Hopfield network was configured to store 101. The positive coupling between units 1 and 3, together with its bias fields, makes 101 the unique lowest-energy state.

E(s) = -s1s3 - 0.5s1 + s2 - 0.5s3

Click a bit to create your own corrupted input, then run a deterministic sweep. The network moves downhill in its discrete energy landscape.

Current state

Network and local fields

All possible states

101 is the stored attractor.

3. Boltzmann probability

Energy becomes a distribution over states.

For two visible binary units, a positive coupling J favors the jointly active state. The partition function Z normalizes the probabilities over all four configurations.

E(v1,v2) = -J v1v2   |   p(v) = exp[-E(v)/T] / Z
Z = 0.00The state 11 has the lowest energy.

4. Positive and negative phases

Learning compares data correlations with model correlations.

A tiny Boltzmann machine has two visible units and one hidden unit. During the positive phase, visible data are clamped to v = (1, 0). During the negative phase, the model is allowed to run freely.

Delta Wi = eta [ <vih>data - <vih>model ]
Delta ai = eta [ <vi>data - <vi>model ],   Delta b = eta [ <h>data - <h>model ]

The model averages are evaluated exactly over eight states here; each click uses eta = 0.7. A realistic Boltzmann machine instead estimates them by sampling.

v1
1
h
v2
0

Positive phase

<v h>data = (0.500, 0.000)visible units fixed to the training pattern

Negative phase

<v h>model = (0.250, 0.250)all states contribute according to p(v, h)

What is being trained

Visible-state distribution

teal: model p(v); coral: empirical data pdata(v)

W1 = +0.000W2 = +0.000a1 = +0.000a2 = +0.000b = +0.000pmodel(1,0) = 0.250

The model initially treats all visible patterns equally.

5. Hidden units

A hidden unit can represent a feature shared by visible data.

Visible units are the observed data. Hidden units are not extra measurements: their activations are inferred from the visible configuration. In this hand-designed toy, h1 is likely when both endpoints are active, while h2 is likely for an isolated active middle unit.

p(hj = 1 | v) = 1 / [1 + exp(-bj - sum(i) Wijvi)]

The weights here are chosen to make the mechanism inspectable, not fitted from data. A trained RBM would discover whatever hidden features best explain its training distribution.

Visible pattern v

Inferred hidden features

For v = 101, the endpoint feature h1 is much more likely than the middle feature h2.