A student network can learn to read handwritten digits without being shown a single digit. It never sees an image of one. It never sees a label. Its whole training diet is random static, and the only thing it is asked to do is copy a few extra numbers that a teacher network produces.
Then you test it on real digits, and it gets most of them right.
This is the MNIST experiment from the subliminal learning paper, written by researchers at Anthropic, UC Berkeley, and other institutions [1]. In their version, the student reached over 50% accuracy on the MNIST test set [1]. I rebuilt it in plain NumPy to see it for myself and to find out what it depends on. It runs in about 20 minutes on a free Colab CPU, and all the code is below.
There is a catch, and it is the more interesting half of the story. The effect only appears when the student and the teacher start from the same random weights. Give the student a different random start and nothing is transferred. Most of this article is about why.
The setup
Take a small multilayer perceptron: 784 inputs, two hidden layers of 256 units, and 13 outputs. The first 10 outputs are the digit logits, one per class. The last 3 are what the paper calls auxiliary logits: extra outputs that exist only so there is something to copy [1].
The recipe has four steps. First, draw random weights and call them the starting weights. Second, make a teacher by copying them and training on MNIST [4] for five epochs, with a cross-entropy loss on the 10 digit outputs only. The 3 extra outputs get no loss and no labels. They just change as a side effect of the shared layers changing. Third, make a student by copying the same starting weights again. Train it on random Gaussian noise with one job: match the teacher’s 3 extra outputs on that same noise. This is knowledge distillation [3], with the digits removed. Fourth, test the student’s 10 digit outputs on real MNIST test images.
One detail is worth pausing on. The student’s loss never touches the 10 digit outputs, so the weights that produce them receive exactly zero gradient and stay at their random starting values for the whole run. Whatever the student learns, it learns in the layers underneath.
Pixels are divided by 255 and then standardised, so real digits and unit-variance noise sit on a similar scale.
The code
Everything is NumPy: a forward pass, a hand-written backward pass, and SGD with momentum. Before trusting anything else I checked the backward pass against finite differences, and the worst relative error across both losses was about 2e-8.
First the network and its gradients.
import numpy as npdef init_params(seed, sizes):rng = np.random.default_rng(seed)p = []for i in range(len(sizes) - 1):p += [rng.normal(0, np.sqrt(2 / sizes[i]), (sizes[i], sizes[i + 1])).astype(np.float32),np.zeros(sizes[i + 1], np.float32)]return pdef forward(p, X):h1 = np.maximum(X @ p[0] + p[1], 0)h2 = np.maximum(h1 @ p[2] + p[3], 0)return h2 @ p[4] + p[5], (X, h1, h2)def backward(p, cache, dout):X, h1, h2 = cached2 = (dout @ p[4].T) * (h2 > 0)d1 = (d2 @ p[2].T) * (h1 > 0)return [X.T @ d1, d1.sum(0), h1.T @ d2, d2.sum(0), h2.T @ dout, dout.sum(0)]The two losses are where the setup lives. Each one writes zeros into the columns it is not supposed to train.
K = 10 # the first 10 outputs are digit logits, the rest are the extra onesdef ce_loss_grad(out, y):z = out[:, :K] - out[:, :K].max(1, keepdims=True)pr = np.exp(z) / np.exp(z).sum(1, keepdims=True)d = np.zeros_like(out)d[:, :K] = prd[np.arange(len(y)), y] -= 1return d / len(y) # zero gradient on the extra outputsdef aux_loss_grad(out, target):d = np.zeros_like(out)d[:, K:] = 2 * (out[:, K:] - target) / len(out)return d # zero gradient on the 10 digit outputsNow the teacher and the student. The student’s loop never loads a single MNIST image.
def sgd_step(p, v, g, lr, mu=0.9):for a, vi, gi in zip(p, v, g):vi *= muvi -= lr * gia += videf train_teacher(p0, X, y, epochs=5, lr=0.02, bs=128, seed=0):p = [a.copy() for a in p0]v = [np.zeros_like(a) for a in p]rng = np.random.default_rng(seed)for _ in range(epochs):idx = rng.permutation(len(X))for i in range(0, len(idx) - bs + 1, bs):b = idx[i:i + bs]out, cache = forward(p, X[b])sgd_step(p, v, backward(p, cache, ce_loss_grad(out, y[b])), lr)return pdef train_student_on_noise(start, teacher, steps=2340, lr=0.01, bs=128, seed=1):s = [a.copy() for a in start]v = [np.zeros_like(a) for a in s]rng = np.random.default_rng(seed)for _ in range(steps):noise = rng.normal(0, 1, (bs, s[0].shape[0])).astype(np.float32)target = forward(teacher, noise)[0][:, K:] # the teacher's extra outputsout, cache = forward(s, noise)sgd_step(s, v, backward(s, cache, aux_loss_grad(out, target)), lr)return sAnd the experiment itself.
sizes = [784, 256, 256, 13] # 10 digit outputs + 3 extrap0 = init_params(0, sizes) # the shared starting weightsteacher = train_teacher(p0, X_train, y_train)student = train_student_on_noise(p0, teacher) # same startother = train_student_on_noise(init_params(1000, sizes), teacher) # different startdef accuracy(p, X, y):return (forward(p, X)[0][:, :K].argmax(1) == y).mean()What happened
I ran everything for 5 seeds, each with its own starting weights, teacher and noise. The settings were not tuned: one learning rate, one number of steps.
The teacher reaches 97.8% on the test set, which is ordinary for a network like this. The untrained starting weights score 9.1%, about chance for ten classes. The student that started from the same weights and saw only noise reaches 89.3%, with individual seeds between 81.1% and 93.8%. After the first 234 of its 2,340 steps, a tenth of the run, it was already at 43.4% on average, and the average crossed 50% by step 468.
A result like this deserves suspicion, so there are two controls.
For the first, I gave the student different random starting weights and changed nothing else: same teacher, same noise, same loss. It ended at 8.8% on average, and its best seed reached 13.2%. That is chance.
For the second, I kept the shared starting weights but trained the teacher on shuffled labels. After five epochs that teacher had fit its own shuffled labels to only 16.6% training accuracy, so it had barely learned anything. The student copying it scored 11.0%. This is a weaker control than it sounds. It shows that a teacher with nothing real to pass on passes nothing on. It does not show what a teacher that had memorised the noise would do.
Together the controls narrow the effect down: it needs a teacher that actually learned digits, and a student that started from the same place.
Here is the argument that made it click for me. The paper proves a result in this spirit [1]. What follows is my own first-order sketch, not their proof, so read it as intuition and not as a citation.
Let f(x; θ) be the vector of the 3 extra outputs for weights θ and input x. The teacher’s weights are the starting weights θ₀ plus a change δ, so θ_T = θ₀ + δ. The student minimises the squared gap between its extra outputs and the teacher’s, averaged over noise inputs:
L
(
θ
)
=
1
2
E
x
∥
f
(
x
;
θ
)
−
f
(
x
;
θ
T
)
∥
2
L(theta)=tfrac{1}{2},mathbb{E}_{x},lVert f(x;theta)-f(x;theta_T)rVert^{2}
My code drops the ½, which only rescales the learning rate. Let J(x) be the Jacobian of f with respect to θ at θ₀. When δ is small, the teacher’s outputs are approximately f(x; θ₀) + J(x)δ, so the gradient at the student’s starting point is:
∇
L
(
θ
0
)
≈
−
E
[
J
⊤
J
]
δ
=
−
G
δ
nabla L(theta_0)approx-,mathbb{E}!left[J^{top}Jright]delta=-G,delta
where G = E[JᵀJ] is an average of matrices of the form JᵀJ, so it is positive semidefinite. The student’s first step is η Gδ for a learning rate η, and its overlap with the teacher’s change is:
⟨
Δ
θ
,
δ
⟩
=
η
δ
⊤
G
δ
=
η
E
∥
J
(
x
)
δ
∥
2
≥
0
langle Deltatheta,,deltarangle=eta,delta^{top}G,delta=eta,mathbb{E}lVert J(x)deltarVert^{2} ge 0
In words: the student’s first step cannot point away from the direction the teacher moved, and nothing in the argument cares what x is. Noise works as well as anything.
Where does the shared start enter? In two places. δ is measured from θ₀, and the student’s Jacobian is evaluated at θ₀ as well, so the two live in the same coordinates. Start the student somewhere else and δ becomes a displacement from a point the student never visited. The argument then says nothing about it.
The sketch is first order and about one small step. My student takes 2,340 steps with momentum, which is not small. So I measured the thing directly: the cosine between how far the student’s two hidden-layer weight matrices moved and how far the teacher’s moved. For the same-start students it averaged 0.72. For the different-start students it averaged 0.02. For those I compared the student’s change with the teacher’s actual change, even though the two started from different points.
I do not have a derivation for the last step, from “the hidden layers moved the way the teacher’s did” to “the fixed random digit outputs now read digits correctly.” I am reporting what I measured. A follow-up study on this same MNIST setup tracked gradient alignment during training and found it stays weakly positive throughout [2]. That fits the picture, but I would not call it settled.
How little does the student need?
I started with 3 extra outputs, as in the paper’s setup [1]. Then I changed the number, from 1 to 10, with 3 seeds each. With a single extra output the student reached 78.7% on average. With ten it reached 90.8%.
That is one number per noise image, with no digit label anywhere in it, and the student still moves from chance to close to 80%. The line dips at 5 extra outputs, but that is one seed at 53.4% while the other two are above 92%. Three seeds per setting is too few to read a trend between 2 and 10, so I will only claim the first part.
What this does and does not show
The language-model version of this result is the one that drew attention. A teacher model prompted to love owls generates sequences of numbers, and a student fine-tuned on those numbers picks up the preference, even after the data is filtered [1]. The effect appeared only when teacher and student shared the same base model [1]. The toy here has the same ingredient: a shared starting point, and nothing in the training data that mentions the trait.
What the toy cannot tell you is whether language models do it for the same reason. The student here imitates continuous outputs, not sampled text. It is one small network, one dataset, a handful of seeds, and my own choice of loss and settings. A mechanism that holds for a first-order step in an MLP may not carry over to a transformer.
What it does give is a clean place to poke at the idea. The practical point from the paper stays on the table: if you train on another model’s outputs, and that model shares your starting point, filtering the data for what it looks like may not be enough to keep out what it carries [1].
The thing I keep coming back to is the student’s digit readout. Not one weight in it was ever trained. Whatever the student learned, it learned in layers that began as an exact copy of the teacher’s, and that turned out to be enough.
References
[1] A. Cloud, M. Le, J. Chua, J. Betley, A. Sztyber-Betley, J. Hilton, S. Marks, O. Evans, Subliminal Learning: Language Models Transmit Behavioral Traits, arXiv:2507.14805
[2] C. Kitkana, Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment (2026), ICLR 2026
[3] G. Hinton, O. Vinyals, J. Dean, Distilling the Knowledge in a Neural Network (2015), arXiv:1503.02531
[4] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-Based Learning Applied to Document Recognition (1998), Proceedings of the IEEE
