A Held-Out Site Is a Sample of One: What Random, Spatial and Site-Disjoint Splits Each Estimate, Why Leakage That Differs Between Configurations Reorders a Comparison, and How Many Sites a General Claim Needs

Abstract

Life of Research

A September 2026 study of farm-machinery traversability reports that a forward-facing near-infrared camera appears to beat a colour camera on a paddy-boundary class under its corpus's own sequence-level split, that the advantage disappears once recording sites are held apart, that the near-infrared configuration gained more from having seen a site than colour did, and that two leads the second camera held at one held-out site changed sign when the held-out site was rotated. Each of those findings touches a literature that disagrees with itself: ecologists and remote-sensing researchers dispute whether spatial cross-validation corrects an optimistic bias or introduces a pessimistic one, clinical machine learning had the same dispute over subject-wise splitting, and machine-learning benchmarks report both that rankings survive distribution shift and that they do not. In this analysis the author reviews and synthesises published results to show that these disputes concern three different questions an evaluation can be asked - how accurate a model is at a stated target (level), which of two configurations is better there (order), and whether an advantage holds across the population of sites a deployment will meet (generality) - and that each needs a different design. The level dispute resolves once the prediction target is specified, and the published positions on both sides agree on that resolution more than their titles suggest. The order question has a failure mode the level dispute does not cover: when leakage inflates configurations unequally, a split that shares sites can reverse a comparison, and six published results in four fields show it doing so or coming close: reversals in Kang (2026), Zech et al. (2018), Liang et al. (2017) and, on small margins, Meyer et al. (2019) and Schratz et al. (2018), and a several-fold shrinkage in Milioto et al. (2018). The generality question is a small-sample problem in sites, not in images: one held-out site is one observation. The paper consolidates 24 published results in a table, derives the condition under which leakage preserves an ordering, states a procedure that assigns each evaluation the claim it licenses, and specifies five studies that would settle what remains open.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 124 papers by Pranay Mahendrakar, or his ORCID record.