Publication update, 22 July 2026: This work is now published open access in Chemical Science. I have updated this post to reflect the final, peer-reviewed paper.

A benchmark is a shared exam

In machine learning, a benchmark is a little like a shared exam. Researchers train their models on one part of a dataset and test them on a separate part. Because everyone answers the same questions, the resulting scores give us a convenient way to compare methods—or at least that is the idea.

Now imagine that the exam contains questions that already appeared in the homework. Or that the same question appears twice with two different correct answers. A student might score unusually well by recognising familiar material, or lose points because the answer key contradicts itself. Either way, the score no longer measures quite what we thought it did.

The same problem can occur in molecular machine learning. A test molecule may also be present in the training data. Nearly identical molecules can sit on both sides of the split. The same compound can carry conflicting measurements. In drug–target interaction datasets, the compounds may differ while the protein targets are identical or extremely similar. A model can then succeed through familiarity rather than by learning a rule that transfers to genuinely new chemistry or biology.

That possibility made us want to inspect the exam before designing yet another student.

What we audited

We built an open-source toolkit called BenchAudit and applied it to 51 dataset configurations from Polaris, Therapeutics Data Commons, MoleculeNet, and several widely used drug–target interaction benchmarks. Together, these collections have accumulated more than 10,000 citations, so the question is not academic: they underpin a great deal of work in the field.

The audit follows the data from basic hygiene to the final train–test split. It looks for duplicate molecules, contamination between splits, conflicting labels, highly similar structures, and sharp changes in activity between near-neighbours. For drug–target interaction tasks, it also checks whether proteins or complete drug–target pairs recur across splits and how similar the protein sequences are.

These checks are diagnostic, not automatic verdicts. A shared molecular scaffold can be scientifically meaningful. The same drug tested against two different proteins is not a duplicate experiment. Activity cliffs—similar molecules with very different activities—are often exactly the difficult cases a useful model should learn. The point of an audit is to make these properties visible, so that we can decide whether a split actually tests the kind of generalisation being claimed.

What we found

Problems were widespread, but they did not look the same everywhere. We found cross-split contamination, label conflicts, and substantial structural redundancy across many of the benchmarks. The drug–target interaction datasets deserve particular care: exact duplicate drug–target pairs were comparatively rare, yet the same or closely related proteins and ligands often appeared across training and test data. A model can therefore be evaluated in a very familiar neighbourhood even when the complete interaction pair is technically new.

We then tested whether these issues really move the numbers. In controlled experiments, we deliberately added different kinds of noise to 41 datasets. Conflicting or random labels made performance worse, while train–test contamination made it look better. The direction of the effect was consistent across random forests, gradient-boosted trees, and a neural network. In other words, these are not merely untidy rows in a spreadsheet; they can change the conclusion of an experiment.

The final paper adds another test that I find especially informative. We kept every model and every prediction fixed, but changed the composition of the test panel. The original leading model remained first on audit-clean panels in 23 of 28 datasets, so the result is not simply that every winner benefited from obvious leakage. Yet rankings became much less stable when test panels contained more near-training examples or conflicting labels. In the 14 datasets where a conflict-enriched panel could be constructed, the original leader’s chance of remaining first fell below 50% in 12 cases.

That is a more useful conclusion than saying that every leaderboard is wrong. A leading model may well be the best model for a particular benchmark. But its lead is conditional on what the test set contains, and small differences in a table should not automatically be treated as evidence of a generally superior method.

What I take from it

A benchmark score is never a property of the model alone. It belongs to the model, the dataset, the split, and the chemical or biological composition of the test set. If a paper makes a claim about new scaffolds, unseen targets, or real-world generalisation, the benchmark has to create that challenge explicitly.

For me, this moves dataset auditing from a cleanup step to part of the scientific method. We should report what is shared across splits, explain what kind of novelty the test set represents, and check whether a model ranking survives on clean or deliberately challenging subsets. Better documentation helps, but provenance alone does not guarantee quality; even a reproducible split can reproducibly test the wrong thing.

Benchmarks are still indispensable. They let us build on each other’s work and measure progress without repeating every experiment from scratch. We just need to stop treating them as neutral yardsticks. As Pat Walters has argued, the field needs to look beyond the bold numbers in a table. The data beneath those numbers are part of the result.

Read the paper

Schuh, M. G.; Daniluk, A.; Sieber, S. A. “Auditing widely used biomolecular benchmarks reveals systematic data inconsistencies.” Chemical Science 2026. https://doi.org/10.1039/D6SC01799A

The BenchAudit source code and documentation are openly available.