Campaign evaluation

What a model-guided campaign actually buys, measured rather than asserted. Every figure on this page is derived from output/campaign*.tsv at request time, so it cannot drift from the data behind it.

Reads split into two independent halves: the arm trains on one and is scored on the other.

What was measured

The replay runs the experiment in a real campaign's order. A pool is sampled from the library, 96 variants are measured as a seed, and then each arm spends 96 measurements per round and retrains, for 8 rounds — 864 measurements in all, repeated 50 times independently. Arms are only comparable at the same spend, which the design guarantees.

Scored on reads the arm never saw. The counts are split binomially into two independent half-depth replicates; each arm trains and self-scores on one half, and the column reported here (recall_top100_replicate) is measured on the other. Nothing below is an arm re-counting noise it had already fitted.

What the campaign bought

Bad draw is how often a single campaign ends below half of what the best arm typically returns (here, below 2 of the top 100). It is the number that decides whether you are willing to run this once.

Median over repeats. Blue is the CNN, orange the additive ridge, grey the model-free arms; the dashed grey line is the zero-shot language-model score, which flattens once its fixed ranking is spent — a frozen score has nothing to update.

Where the gain comes from

There is no decomposition to report on this dataset. The rungs are not monotone — greedy scores below the rung beneath it — so forcing shares out of them would print contributions above 100% and below zero. That the ordering breaks is itself the finding: see the decision rule below.

What does not survive re-measurement

The single best variant each arm found, scored again on the independent replicate. This is the column that decides whether a headline number can be quoted.

"The model found a +5.41 variant" cannot be reported. Re-measure it and every arm lands within a few tenths of the others — the model's pick and a random pick are not distinguishable on this metric. Three things keep that from meaning "no benefit". First, the test has no power: with 50 repeats the standard error of the difference is around 0.17, so a real 0.3-unit gap would go unseen, and failing to reject is not equivalence. Second, the shrunk column does separate — the diversity-capped and PLM-input arms clear random by more than 2σ. Third, not all of the curse belongs to the model: the random arm incurs 1.04 of it simply by taking a maximum over many noisy values.

One caveat of our own: splitting the reads halves the depth, so selection here runs on twice the noise of the real assay. The curse at full depth is smaller — roughly by √2. The direction is certain; this magnitude should not be quoted as-is.

A failing campaign is visible early

2 of 50 repeats of the CNN arm ended below half the typical haul. Recall needs the answer key, so no running campaign can read it — but the mean score of the batch just measured is free, and the two groups separate on it nowhere in this table.

That turns a 4% chance of a wasted campaign into a small loss, since a campaign in the failure mode can be stopped and reseeded. With only 2 failures this is suggestive rather than established — but the mechanism is plain: a poor seed leaves the model nothing to learn, and the batches it buys go bad immediately.

When this works, and when it does not

What the efficiency depends on

The multiple does not transfer between datasets, and the spread is not small: the same code, the same budget and the same protocol return anything from 2× to 45× depending only on which library it is pointed at. So the first question about a new target is which end of that range it sits at.

Part of the answer is structural and can be read off before paying for anything. gb1 is the complete double-mutant library — every single substitution, and 99.8% of every possible pair of them — which is a different kind of object from the rest, and it is the one that returns 45×. But density alone does not order the others. ube4b holds 2.9% of its possible doubles and returns 21×, while pab1 holds 4.0% and returns 2×. Neither density nor the z of the target cut predicts the four non-gb1 results, so we can currently bound the question, not answer it: a combinatorially complete library is a good bet, a sparse one is a coin toss that has to be measured.

On pab1 the ordering inverts: the additive ridge beats the network and both sit near the random floor. That is the case to watch for — not a weaker version of the gb1 result but a different one, where the extra capacity costs accuracy. Which regime a library is in is, on this evidence, something to establish with a retrospective replay before committing an assay budget to it.

The four non-gb1 rows above come from 10-repeat first-pass tables except pab1, so their multiples carry more spread than gb1's. Longer runs are in flight.

Read depth, and the hole it found

Pricing a campaign means first checking that the target exists. All five dataset files here are byte-identical to the published source, with no residue mismatches, no duplicate variants and no split overlap. Then there is this.

A quarter of the bgl3 library was never measured, and those rows hold 99 of its top 100 scores. Every selection metric computed on unfiltered bgl3 was therefore scoring the recovery of rows nobody measured. Re-running with a five-read threshold moves the two halves in opposite directions, and both are corrections: the position-held-out correlation rises from 0.168 to 0.406, because a quarter of the target was not a function of the sequence at all, while hit@100 falls from 0.836 to 0.120, because the unfiltered top hundred was a large tied block of one arithmetic value and recovering it was an easy ranking problem.

This is not our bug. The file is the published one, and the counts come from the original authors' alignment of the raw reads, so it applies to the bgl3 results in the literature as much as to ours. gb1 has the milder version: nothing unmeasured, but its top hundred sits at a median of 15 reads against a library median of 249 — the winner's curse, visible in the raw counts before any model is fitted.

What we do

Evaluation. Not making models bigger — working out which numbers can actually be used. This page is itself the output of that work: the bgl3 selection metrics we first reported were scoring unmeasured rows, the noise ceiling we first reported was wrong twice, and the best variant we first reported was indistinguishable from a random pick once re-measured. All three were found by attacking our own results.

• Measurement noise ceilings. Split the reads into replicates and correct back with Spearman–Brown, giving the highest correlation any predictor can reach against these scores. Settles whether more tuning is worth paying for or is fitting noise.
• Data audits. Read depth, pseudocount artifacts, tie structure, wild-type consistency, split overlap, and a byte comparison against the published copy. The bgl3 hole came out of this.
• Campaign replay. What model-guided selection returns against random, additive and zero-shot baselines at equal spend, scored on an independent replicate.
• Pricing the winner's curse. Separate selection bias from the reported improvement, and give the shrinkage-corrected value.
• Split protocol review. Whether reported performance depends on position leakage, and what it becomes under real extrapolation.

References

Starting papers

1. Sam Gelman, Sarah A. Fahlberg, Pete Heinzelman, Philip A. Romero, Anthony Gitter. Neural networks to learn protein sequence–function relationships from deep mutational scanning data. PNAS 118(48), e2104878118, 2021. doi:10.1073/pnas.2104878118 — code and data: gitter-lab/nn4dms
2. Wen Torng, Russ B. Altman. 3D deep convolutional neural networks for amino acid environment similarity analysis. BMC Bioinformatics 18, 302, 2017. doi:10.1186/s12859-017-1702-0

Dataset sources

1. C. Anders Olson, Nicholas C. Wu, Ren Sun. A comprehensive biophysical description of pairwise epistasis throughout an entire protein domain. Current Biology 24(22), 2643–2651, 2014. — gb1
2. Karen S. Sarkisyan et al. Local fitness landscape of the green fluorescent protein. Nature 533, 397–401, 2016. — avgfp
3. Daniel Melamed, David L. Young, Caitlin E. Gamble, Christina R. Miller, Stanley Fields. Deep mutational scanning of an RRM domain of the Saccharomyces cerevisiae poly(A)-binding protein. RNA 19(11), 1537–1551, 2013. — pab1
4. Lea M. Starita et al. Activity-enhancing mutations in an E3 ubiquitin ligase identified by high-throughput mutagenesis. PNAS 110(14), E1263–E1272, 2013. — ube4b
5. Philip A. Romero, Tuan M. Tran, Adam R. Abate. Dissecting enzyme function with microfluidic-based deep mutational scanning. PNAS 112(23), 7159–7164, 2015. — bgl3. The read counts were produced by the nn4dms authors from the raw SRA reads, and are the origin of the zero-read rows above.

Methods

1. Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, Alexander Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. NeurIPS 2021. — the masked-marginal protocol
2. Zeming Lin et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 379(6637), 1123–1130, 2023. — ESM-2
3. Pascal Notin et al. ProteinGym: large-scale benchmarks for protein fitness prediction and design. NeurIPS 2023. — the supervised split protocol (random / modulo / contiguous)
4. Shuzhe Kawashima et al. AAindex: amino acid index database. genome.jp/aaindex

Prior work we cite rather than claim

We rediscovered these independently; they are already published.

1. The position-held-out collapse. Barethiya, Huang, Stumpf, Liu, Guan, Chen. PLOS Computational Biology 22(3):e1013728, 2026 — the same five datasets, the same four model families, the same one-hot(21) + AAindex(19) encoding, with positional extrapolation at or below zero.
2. Networks barely beating linear regression is the field's stated consensus. Wittmann, Advances in Machine Learning for Directed Evolution (arXiv:2609.03046, 2026).
3. Model-free baselines being competitive. FLIP2 (Didi et al., ICML 2026) concludes that one-hot ridge cannot generalise to new positions or scaffolds.

Measured on PyTorch, NVIDIA DGX-1 (V100). Campaign replay: eight arms × 50 repeats per dataset. Datasets are all public; the code is a PyTorch reimplementation of nn4dms. Full working notes in FINDINGS.md.