Slide Bag Net
Multiple-instance learning (MIL): an attention network that learns whether a tumour carries a given mutation from slide-level labels only
🎯 Learning objectives
- Show the bag / instance distinction and the MIL assumption on a slide
- Explain why mean, max and attention pooling behave differently on the same data
- Read an attention map and say what it does not show
- Trigger shortcut learning (lab stain learned instead of the mutation) and diagnose it on the test set
- List the points where the simulation departs from a real mutation predictor
The simulation
The simulation is embedded below; open it in a new tab for full screen. The language switch (TR/EN) changes the explanations on the page; the network is the same.
Keys: 1 2 3 steps · space train/pause · N next test slide.
The setting: why “weak” supervision
In molecular pathology the label is always at case level: the sequencing report says “EGFR mutant”, not which cells on the slide betray it. The simulation builds that situation in its smallest form:
- 120 synthetic slides, 16 patches each. 80 for training, 40 held out for testing.
- One label: the slide is mutant or wild-type. No patch is labelled.
- In a mutant slide only 2 to 4 of the 16 patches carry the mutant pattern (large nuclei + cytoplasmic pallor + prominent nucleoli). The rest are ordinary tumour, stroma and look-alikes: pleomorphic patches (large nuclei, no pallor) and degenerate patches (pallor, ordinary nuclei). Wild-type slides contain the same look-alikes.
- Each patch is reduced to six measurements. In a real pipeline this step is a frozen foundation model that turns a 256-pixel tile into 1024+ numbers (Chen et al. 2024; Xu et al. 2024); the simulation uses six so that every weight stays readable.
- Every slide was scanned at one of two labs; lab B’s haematoxylin is weak. By default the lab is independent of the mutation.
The network: three parts
Every modern slide-level model has three parts (Ilse, Tomczak, and Welling 2018; Lu et al. 2021):
- Shared encoder: maps each patch’s six numbers to a hidden vector; the same weights for every patch.
- Pooling: turns the 16 hidden vectors into one slide vector. The simulation offers three choices:
- mean: every patch counts the same; 2 informative patches in 16 are diluted;
- max: per dimension, the most positive patch wins; the MIL assumption as an operator; the gradient reaches one patch per dimension;
- attention: a learned score per patch, normalised with a softmax, then a weighted average. A soft, learned max.
- Classifier: turns the slide vector into P(mutant). Only the slide label reaches the loss; backpropagation distributes the gradient through pooling into the patches.
The Evidence tab lists each patch’s share of the logit (attention weight × the classifier’s view of the patch). Weight and contribution are different numbers: a patch can receive high attention and push the logit down.
Lecture recipe (40 min)
1
Slides. Show the training grid; try to find the mutant slides by eye. The frame colour is the only clue. Open a mutant slide in the inspector and switch on Reveal patch-level truth: 2 to 4 patches in 16. Use the words “bag” and “instance” here.
2
Mean pooling. Switch on Reveal test labels in step 1. On the Train step set pooling to mean and train. The loss stays high and training accuracy stalls around 94%; but the dashed test curve and the AUROC (≈0.95) are good. Mean pooling ranks well and cannot make a confident call. Explain the difference between accuracy and AUROC here.
3
Max. Reset the weights, switch to max. It fits the training set; its test AUROC is the lowest of the three (≈0.88). The gradient reaches one patch per dimension; a single look-alike patch flips a call.
4
Attention. Reset, switch to attention. Watch the heat move onto the right patches of the inspected slide while the loss falls fast; open the Evidence tab. In the encoder weight map, show that the size, pallor and nucleoli columns grow together while the stain column stays near zero. Then set weight decay to 0 and retrain: the loss sticks to zero, training is 100%, test AUROC slips. With 80 slides attention starts to memorise; this is why CLAM uses dropout, weight decay and early stopping on a validation split.
5
Test. Press N forty times. Say the AUROC out loud. Read the confusion matrix.
6
The trap. On the Slides step switch on Confound lab with mutation (90% of mutants from lab B). Reset and retrain with attention. Training accuracy reaches 100%; the attention map flattens; the stain column (x₆) grows; test AUROC falls from ≈0.97 to ≈0.75. The test set has equal numbers of slides from both labs, so the shortcut does not work there.
Reading an attention map
The map shows where the model placed its weight, not where the mutant pattern is. Three things go wrong routinely:
- Attention can be high on a patch that pushes the logit down (weight ≠ contribution).
- Attention can spread over patches that share a slide-wide property (stain) unrelated to biology.
- An unattended patch is not evidence of absence; the softmax has one unit of weight to distribute.
With the lecturer toggle you can compare the map against patch truth; in real data you cannot. That is why published attention maps are read by a pathologist and reported as a sanity check, not presented as a localisation result.
Shortcut learning and multi-site data
The network has two routes to a low training loss: learn the three-way conjunction in a few patches, or read the stain intensity, which is on every patch of the slide, with one weight. Gradient descent takes the cheap route (Geirhos et al. 2020). In TCGA, site-specific stain and scanner signatures have been shown to inflate mutation and survival predictors (Howard et al. 2021); site-stratified splits and stain augmentation are standard for that reason.
From the simulation to a real predictor
The structure stays, the scale changes: tissue detection → tens of thousands of 224–512 px patches at 10–20× → embedding with a frozen foundation model → this attention block, with slide labels from sequencing. Splits by patient, validation on an external cohort. Reported AUROC for the best-studied targets lies roughly between 0.75 and 0.95 depending on target and cohort: MSI in colorectal cancer (Kather et al. 2019; Echle et al. 2020), EGFR in lung adenocarcinoma (Coudray et al. 2018), pan-cancer screens (Kather et al. 2020; Fu et al. 2020). Good enough for pre-screening, not enough to replace the assay.
What the simulation leaves out: 16 patches instead of 20,000; six readable measurements instead of an unreadable 1024-dimensional embedding; one slide per patient; a balanced class ratio (most mutations are rare); probability calibration; and the fact that in a real tumour the “mutant pattern” is a statistical tendency, not a crisp phenotype present at a fixed fraction. For the cures of the lab trap continue with Stain Shift; for the real source of the labels and the class ratios, with the cBioPortal module.
Check yourself
With mean pooling the loss stays high and training accuracy stalls around 94%, yet the test AUROC is good. Why?
With the lab trap on, training accuracy is 100% and test AUROC ≈ 0.5. What does that pair tell you?
A patch receives high weight on the attention map. Does it carry the mutant pattern?
References
Works cited on this page are listed below; for the whole series see References.