Networks and test sets

What this project does, which networks it fits, and how to change them.

Overview

A deep mutational scanning experiment measures thousands of variants of one protein and gives each a score: how well that variant still works. Only a tiny corner of the possible sequences can ever be measured, so these models learn the pattern from the measured variants and predict the score of ones nobody has made yet. That turns the search for a better protein from bench work into a ranking problem.

Each network below takes a variant's amino acid sequence and returns one predicted score; they differ in how much of the sequence they look at once, and whether they use the folded structure. A separate 3D CNN works on the atoms around a single residue instead; its data is on the atom boxes page.

Measured scores on the results page say how closely predictions track measurements overall. That is not the same as being useful for engineering: a model can score well by telling broken variants from working ones and still rank the best candidates poorly, which is what the mutation scan and the percentile column on score variants are for.

Reference networks and test sets

The established baseline settings, one configuration per dataset and model type. A configuration says which measured dataset to learn from, how each variant is turned into numbers, which network to fit, and which variants are held back for testing.

Configuration	Protein	Network	Test set	Batch	Learning rate
reference/avgfp_cnn reference	avgfp	Sequence CNN — conv 128 filters, kernel 3 x5 + dense 100 input 237 residues x 40 features conv1d 128 filters, kernel 3, leaky relu conv1d 128 filters, kernel 3, leaky relu conv1d 128 filters, kernel 3, leaky relu conv1d 128 filters, kernel 3, leaky relu conv1d 128 filters, kernel 3, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	5,403 held back	64	0.001
reference/avgfp_fc reference	avgfp	Fully connected — dense 100 x3 input 237 residues x 40 features flatten dense 100, leaky relu, dropout 0.2 dense 100, leaky relu, dropout 0.2 dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	5,403 held back	32	0.0001
reference/avgfp_gcn reference	avgfp	Graph CNN — graph conv 128 filters x2 + dense 100 input 237 residues x 40 features graph conv 128 filters, leaky relu graph conv 128 filters, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	5,403 held back	32	0.0001
reference/avgfp_lr reference	avgfp	Linear — linear, no hidden layer input 237 residues x 40 features flatten dense 1 (predicted score)	5,403 held back	128	0.0001
reference/bgl3_cnn reference	bgl3	Sequence CNN — conv 32 filters, kernel 17 + dense 100 input 501 residues x 40 features conv1d 32 filters, kernel 17, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	2,666 held back	64	0.0001
reference/bgl3_fc reference	bgl3	Fully connected — dense 100 x2 input 501 residues x 40 features flatten dense 100, leaky relu, dropout 0.2 dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	2,666 held back	32	0.0001
reference/bgl3_gcn reference	bgl3	Graph CNN — graph conv 32 filters + dense 100 input 501 residues x 40 features graph conv 32 filters, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	2,666 held back	128	0.0001
reference/bgl3_lr reference	bgl3	Linear — linear, no hidden layer input 501 residues x 40 features flatten dense 1 (predicted score)	2,666 held back	128	0.0001
reference/gb1_cnn reference	gb1	Sequence CNN — conv 128 filters, kernel 17 x3 + dense 100 input 56 residues x 40 features conv1d 128 filters, kernel 17, leaky relu conv1d 128 filters, kernel 17, leaky relu conv1d 128 filters, kernel 17, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	53,609 held back	32	0.0001
reference/gb1_fc reference	gb1	Fully connected — dense 1000 input 56 residues x 40 features flatten dense 1000, leaky relu, dropout 0.2 dense 1 (predicted score)	53,609 held back	64	0.0001
reference/gb1_gcn reference	gb1	Graph CNN — graph conv 128 filters x5 + dense 100 input 56 residues x 40 features graph conv 128 filters, leaky relu graph conv 128 filters, leaky relu graph conv 128 filters, leaky relu graph conv 128 filters, leaky relu graph conv 128 filters, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	53,609 held back	32	0.0001
reference/gb1_lr reference	gb1	Linear — linear, no hidden layer input 56 residues x 40 features flatten dense 1 (predicted score)	53,609 held back	128	0.0001
reference/pab1_cnn reference	pab1	Sequence CNN — conv 128 filters, kernel 17 x3 + dense 100 input 75 residues x 40 features conv1d 128 filters, kernel 17, leaky relu conv1d 128 filters, kernel 17, leaky relu conv1d 128 filters, kernel 17, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	4,086 held back	128	0.0001
reference/pab1_fc reference	pab1	Fully connected — dense 100 x3 input 75 residues x 40 features flatten dense 100, leaky relu, dropout 0.2 dense 100, leaky relu, dropout 0.2 dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	4,086 held back	128	0.001
reference/pab1_gcn reference	pab1	Graph CNN — graph conv 32 filters + dense 100 input 75 residues x 40 features graph conv 32 filters, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	4,086 held back	128	0.0001
reference/pab1_lr reference	pab1	Linear — linear, no hidden layer input 75 residues x 40 features flatten dense 1 (predicted score)	4,086 held back	128	0.001
reference/ube4b_cnn reference	ube4b	Sequence CNN — conv 128 filters, kernel 3 x5 + dense 100 input 102 residues x 40 features conv1d 128 filters, kernel 3, leaky relu conv1d 128 filters, kernel 3, leaky relu conv1d 128 filters, kernel 3, leaky relu conv1d 128 filters, kernel 3, leaky relu conv1d 128 filters, kernel 3, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	9,830 held back	128	0.0001
reference/ube4b_fc reference	ube4b	Fully connected — dense 100 x3 input 102 residues x 40 features flatten dense 100, leaky relu, dropout 0.2 dense 100, leaky relu, dropout 0.2 dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	9,830 held back	64	0.0001
reference/ube4b_gcn reference	ube4b	Graph CNN — graph conv 128 filters x3 + dense 100 input 102 residues x 40 features graph conv 128 filters, leaky relu graph conv 128 filters, leaky relu graph conv 128 filters, leaky relu flatten dense 100, leaky relu, dropout 0.2 dense 1 (predicted score)	9,830 held back	128	0.0001
reference/ube4b_lr reference	ube4b	Linear — linear, no hidden layer input 102 residues x 40 features flatten dense 1 (predicted score)	9,830 held back	64	0.0001

How to read this

Protein is the deep mutational scanning dataset: thousands of measured variants of one protein. Open a configuration to see how many variants each set holds.
Network is the architecture fitted to the sequences. Click it for the layers. Linear adds up a fixed effect per mutation, so it cannot represent mutations that interact. The other three can, and they differ in how they look at the sequence: a fully connected network sees all positions at once, a sequence CNN slides a window along the chain, and a graph CNN also uses which residues sit close together in the folded structure.
Test set is the group of variants held back from training and only used for the final score, so the numbers above are measured on variants the network never saw. The split is fixed and stored with the data, so every run is scored on the same variants.
Batch and learning rate control the fitting itself, not what the network can represent. They are set per dataset and model.

Edit a configuration

Change a setting to try your own variation. The reference configurations stay read-only so the baseline above is always reproducible; your edit is saved as a new file under configs/custom/, which you then train from the command line.

Start from

What you can change

model — the network. name is one of lr, fcn, cnn or gcn. hidden lists the dense layers, conv lists [filters, kernel] pairs for a sequence CNN, and filters lists the graph layers. A wider or deeper network can fit more, and also overfits more readily on the smaller datasets.
data.split_dir — the test set. Leave the reference split to stay comparable, or remove it and set train_size, valid_size and test_size to split at random instead.
data.encoding — how a variant becomes numbers: one_hot marks which amino acid sits at each position, and aa_index adds 19 measured chemical properties per amino acid.
train — epochs, batch_size, learning_rate, and patience, which stops training once the validation loss has not improved for that many epochs.

Checking a configuration builds the network without training it, so mistakes such as a convolution wider than the protein are reported here rather than failing minutes into a run.

Measured datasets, splits and network designs: Gelman et al. 2021 (sequence models) and Torng and Altman 2017 (atom-box 3D CNN).