A Benchmark Hit Is Not an Exposure: What Published Vision-Language Geolocation Accuracy Says About Location Privacy, Why One Model Locates 61 Percent of One Photo Set to a Kilometre and Fewer Than 12 Percent of Another to a Hundred, and the Exposure Estimand a Privacy Claim Needs
Abstract
Life of Research
Since 2024 a growing literature has argued that general-purpose vision-language models can work out where a photograph was taken well enough to make image geolocation an immediate privacy risk, and a second literature has begun building defences against it. In this analysis the author reviews and synthesises published results to show that the direction of that claim survives the evidence but its usual support does not: the accuracies quoted for it are properties of particular photo sets, adversaries and distance thresholds, and none of them is the quantity a privacy claim is about. The same model places 61 percent of a curated, text-rich stock-photo set within one kilometre, about 23 percent of a public Flickr benchmark, and fewer than 12 percent of sampled street-level images within one hundred kilometres. One video-frame benchmark reports 65.6 percent within a kilometre for clips from one platform and 6.4 percent for clips from another. The obvious deflation, that benchmark images were seen in training, does not explain the capability: on photographs collected by the authors and not drawn from any public source, frontier models still name the city about 60 percent of the time, and freshly taken photographs of Seoul are located about as well as Street View images of the same city. The author separates five variables that published results leave unfixed (image population, harm radius, adversary configuration, counterfactual baseline, and the unit of exposure), shows that defences are mostly scored against a fixed adversary although a published photograph must resist every later one, and proposes an exposure protocol that states each variable before a number is reported.
Questions about this paper
Who wrote "A Benchmark Hit Is Not an Exposure"?
Pranay Mahendrakar wrote "A Benchmark Hit Is Not an Exposure: What Published Vision-Language Geolocation Accuracy Says About Location Privacy, Why One Model Locates 61 Percent of One Photo Set to a Kilometre and Fewer Than 12 Percent of Another to a Hundred, and the Exposure Estimand a Privacy Claim Needs", published 20 Nov 2026. Pranay Mahendrakar is an Indian AI specialist and LLM engineer based in Bengaluru, India. He is the Managing Director of SonyTech, Nodal Coordinator at IIRS-ISRO, and an instructor at Tutorials Point. His work covers large language models, natural language processing, computer vision and retrieval-augmented generation. He publishes open-access research papers and is the author of three books: Just AI With Pranay, Multiverse of AI and It's Me LLM.
Is "A Benchmark Hit Is Not an Exposure" free to read?
Yes. "A Benchmark Hit Is Not an Exposure" by Pranay Mahendrakar is open access under a Creative Commons Attribution 4.0 licence, with the full PDF available from Zenodo at https://doi.org/10.5281/zenodo.24382475. There is no paywall and no account required.
How do I cite "A Benchmark Hit Is Not an Exposure"?
Cite the DOI: Mahendrakar, P. (2026). A Benchmark Hit Is Not an Exposure: What Published Vision-Language Geolocation Accuracy Says About Location Privacy, Why One Model Locates 61 Percent of One Photo Set to a Kilometre and Fewer Than 12 Percent of Another to a Hundred, and the Exposure Estimand a Privacy Claim Needs. Zenodo. https://doi.org/10.5281/zenodo.24382475 A BibTeX entry is provided on this page.