On November 12, 2026, researchers Paarth Batra and Aryan Keluskar from The Biodesign Institute released a preprint on arXiv (2611.16881v1) demonstrating that leading protein language models fail contact map prediction when subjected to invariant amino-acid alphabet relabeling. The diagnostic evaluation reveals that neural architectures rely heavily on canonical residue identities rather than learning invariant coevolutionary statistics, requiring structural biology and computational labs to rethink validation protocols for foundation model predictions.
What happened
Modern single-sequence protein language models (pLMs) frequently predict residue contact maps with high precision. In structural bioinformatics, this accuracy is widely interpreted as evidence that transformers implicitly infer evolutionary constraints like classical multiple sequence alignment models, including pseudolikelihood Potts models and direct coupling analysis. The new study tests this assumption by subjecting leading models to a diagnostic symmetry test: a global bijective permutation of the twenty canonical amino acids.
Because a global bijective relabeling systematically swaps residue symbols across all sequences, all alignment-level information, column conservation, column entropies, inter-column mutual information, and pairwise edit distances remain mathematically identical. Classical inverse-covariance estimators and Potts models provably satisfy this invariance, yielding bit-identical contact scores (max score difference 2 × 10-14) and an exact contact overlap of 1.000. In contrast, every evaluated neural model experienced a collapse in contact prediction, failing to maintain contact maps once the alphabet was relabeled.
Methodology and key findings
The authors evaluated several prominent architectures across diverse scales, including ESM-2 (8M, 35M, 150M, 650M, and 3B parameters), MSA Transformer, and AMPLIFY 120M. They tested three intervention regimes across varying numbers of permuted residue types (nominal parameter m): random derangements, in-class permutations restricted to BLOSUM62 biochemical similarity groups (IVLM, KR, DE, NQ, ST, FYW), and cross-class permutations moving residues outside their biochemical groups.
Under a complete random permutation of all twenty amino acids, ESM-2 650M retained an average contact-set overlap of only 0.015 across nine benchmark proteins, essentially matching chance (such as 0.025 on ubiquitin). MSA Transformer dropped to an overlap of 0.147, and AMPLIFY 120M fell to 0.056. By contrast, when permutations were restricted within biochemical classes (moving 15 residue types), ESM-2 650M preserved 0.655 of its top long-range contacts. Cross-class permutations caused contacts to collapse even faster than uniform random relabeling at intermediate levels, dropping to 0.027 overlap at m = 10 compared to 0.100 for random permutations.
To confirm that degradation was not driven by sequence edit distance, the authors tested a matched-Hamming distance control on cytochrome c. Holding mutated positions identical at 76 of 104 residues, in-class relabeling retained 54 times more contacts (0.699 overlap) than matched uniform random substitutions (0.013 overlap). Furthermore, scaling ESM-2 by 375 times—from 8 million to 3 billion parameters—improved tolerance to in-class biochemical swaps but provided zero recovery under random alphabet derangement. The findings demonstrate what the authors describe as alphabet anchoring: models depend on explicit residue identities rather than computing pure coevolutionary column statistics.
Why it matters for research practice
These findings carry immediate ramifications for computational reproducibility in bioinformatics and machine learning. High benchmark accuracy has often been conflated with mechanistic understanding. When a model succeeds by anchoring to specific token identities rather than learning underlying evolutionary principles, it remains vulnerable to distribution shifts and novel sequence families where canonical patterns diverge.
For research teams designing reproducible AI experiments, the study highlights the necessity of perturbation-based diagnostic tests. Standard benchmarks often reward superficial pattern matching. Introducing invariant transformations—perturbations where ground-truth physics or statistics are provably invariant—provides a rigorous methodology to expose shortcut learning and black-box representation biases before models are deployed in production pipelines.
What changes for researchers
Bioinformatics practitioners and structural biologists should adapt their workflows in three specific ways:
- Maintain classical baselines: Do not treat alignment-based Potts models and direct coupling analysis as obsolete. Because classical estimators remain invariant to alphabet permutations, they provide an orthogonal sanity check against neural hallucinations in contact map prediction.
- Audit foundation models for alphabet anchoring: Teams applying protein language models to de novo protein engineering or deep mutational scanning should incorporate diagnostic symmetry checks into model validation suites to quantify dependence on canonical residue embeddings.
- Separate benchmark precision from mechanistic claims: Reviewers and research evaluators should scrutinize claims that deep learning models implicitly discover natural evolutionary laws without verifying invariance under symmetry-preserving controls.
Sources
- Batra, P., and Keluskar, A. (2026). Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling. arXiv preprint arXiv:2611.16881v1. https://arxiv.org/abs/2611.16881
