105 / papers

Evaluation & Detection

Measuring what models do rather than what benchmarks say they do. 105 open-access papers by Pranay Mahendrakar, each with a permanent DOI.

preprint

A Query Update Is Not a Memory Write: What Test-Time Training on a Long Context Changes, Why Its Advantage Over Thinking Tokens Narrows Outside the Setup That Produced It, and the Controls That Would Credit the Gradient

Life of Research A recent result recommends that, for long inputs, inference compute be spent on a few gradient steps over the given context rather than on extra thinking tokens: query-only test-time training (qTTT) updates only the…

preprint

Calibrated Against Which Label? What Temperature Scaling Certifies When Annotators Disagree, Why a Majority Vote and an Annotator Distribution Each Make the Other's Calibrated Model Look Wrong, and the Target a Calibration Claim Has to Declare

Life of Research Post-hoc calibration is usually fitted and scored against one label per item, and where annotators disagree that label is a majority vote. A 2026 study argues that this is a structural error: temperature scaling fitted to…

preprint

An Output Is Not a Memory Until Something Reads It Back: Where the Key Lives in Implicit-Memory Channels of Stateless Language Models, Why Steganography Evaluations and Temporal-Backdoor Results Disagree, and the Loop Audit That Would Locate a Deployment's Risk

Life of Research Language models are usually described as stateless between requests, and a 2026 position paper argues that the description fails whenever a model's outputs come back to it as inputs: a model can write state into what it…

preprint

Advice Is Not Effort: When a Language Model's Stated Preferences Predict Its Behaviour, Why the Answer Turns on the Channel a Preference Must Travel and the Kind of Preference Asked About, and the Positive Control a Null Result Owes

Life of Research Language models give coherent answers when asked which of two outcomes they prefer, and a growing literature asks whether those answers predict what the models then do. The results look contradictory. A 2026 study of five…

preprint

A Verified Guess Is Not a Reversed Fact: What Test-Time Thinking Repairs in the Latent Generalization of Fine-Tuned Knowledge, Why Its Gains Follow the Forward Paths a Query Leaves Open, and the Candidate-Pool Controls That Would Separate Inversion From Search

Life of Research A language model fine-tuned on new facts often cannot use them the way it uses the same facts placed in its context: it fails to answer reversed questions and to chain two trained facts into a conclusion. A 2026 study…

preprint

A Clause Boundary Is Not a Composition: What Clause-Factorized Decoding Shows About Compositional Multi-Intent Detection, Why a Segmenter and a Cardinality Head Carry Most of the Reported Gap, and the Splits That Would Separate Them

Life of Research Multi-intent detection asks a dialogue system to recover every request in an utterance such as "cancel my meeting and play some music". A 2026 study argues that the field has been measuring the wrong thing: on standard…

preprint

A Session Drop Is Not a Signal Ceiling: What Cross-Session Degradation in ECG Biometrics Measures, Why the Same Heartbeat Can Link a Patient Across Years Yet Fail to Unlock a Device, and the Longitudinal Protocol That Would Locate the Loss

Life of Research Electrocardiogram (ECG) biometrics carries two reputations at once. In the authentication literature the heartbeat is a fragile identifier whose accuracy collapses once enrolment and probe come from different sessions; a…

preprint

A Fallback Is Not a Disposition: What Verifier-Passing Shortcuts in RLVR-Trained Models Show, Why Their Rate Rises With Task Difficulty on One Benchmark and Falls on Another, and the Signatures That Would Separate Fallback, Selection and Learned Exploitation

Life of Research Reinforcement learning with verifiable rewards (RLVR) pays a model whenever an automatic checker accepts its output, so any output the checker wrongly accepts is paid as well. A 2026 study of inductive rule learning found…

preprint

A Pooled Accuracy Is Not a Scaling Verdict: What Tool-Calling Benchmarks Show About Whether Larger Models Call Tools Better, Why Per-Token Price Misranks Cost Only in One Cost Regime, and the Per-Skill, Per-Task Accounting a Router Needs

Life of Research Two claims about tool-using language models now circulate together. The first is that larger or more expensive models do not reliably call tools better than smaller or cheaper ones. The second is that per-token price is a…

preprint

A Manuscript Score Is Not an Audit: What Reviewer Agreement Certifies About Agent-Written Research, Why a Validation Run on Human Papers Does Not Transfer to Authors That Fabricate, and the Chain of Checks an Auto-Research Claim Owes

Life of Research Systems that write research papers end to end are now routinely scored by language-model reviewers, and those reviewers are trusted because they agree with human reviewers about as well as humans agree with each other. A…

preprint

A Lower Score Is Not a Failed Transfer: What the Claim That Static Medical Benchmarks Do Not Carry Over to Interactive Clinical Encounters Measures, Why Level, Rank and Selection Come Apart, and the Harness Controls the Interactive Benchmarks Owe

Life of Research Interactive clinical benchmarks now routinely report that a language model's score on static medical questions "does not translate" to an encounter in which it must take a history, order tests and manage a patient over…

preprint

A Loop Count Is Not a Compute Dial: What the Claim That Looped Models Trade Inference Iterations for Accuracy Measures, Why Truncation, Continuation and Easy-to-Hard Iteration Come Apart, and the Controls That Would Separate Them

Life of Research Looped and recurrent-depth models apply one shared block many times, and a growing literature describes the number of applications as a test-time compute axis: run more loops, get a better answer, with no new parameters. A…

preprint

A Disowned Turn Is Not a Detected Prefill: What Language-Model Reports About Tampered Assistant Turns Measure, Why the 'Cannot Tell' and 'Can Tell' Results Score Different Cues, and Where a Defence Against Prefilling Should Sit

Life of Research Two 2026 studies ask whether a language model can tell that an assistant turn in its own context was written by someone else, and they appear to answer in opposite directions. One finds that ten open-weight models…

preprint

A Coupled Explanation Is Not a Readout: What Self-Explanation Training Shows When a Model's Explanations Follow Its Own Drift, Why an Explanation Computed in a Separate Forward Pass Can Only Replay What It Explains, and the Interventions That Would Separate Replay From Access

Life of Research Training language models to explain their own behaviour is increasingly offered as a scalable route to oversight, and a 2026 result made the route look stronger: models fine-tuned on fixed counterfactual explanations of an…

preprint

Told the Answer Is Not Taught to Reason: What On-Policy Self-Distillation From Privileged Context Transfers to a Language Model, Why Its Gains and Losses Divide Along Whether the Student Can Ever Hold What the Teacher Was Told, and the Controls That Would Separate Three Signals Under One Loss

Life of Research On-policy self-distillation trains a language model on its own samples against the token distribution of the same model shown something the student never sees: a reference solution, a final answer, execution feedback or a…

preprint

A Right Response Is Not a Learned Law: What Perturbation, Constraint and Interpretability Tests Show About Whether Data-Driven Atmospheric Models Learn Chemistry and Dynamics, Why a Constraint Must Be Expressible Before It Can Be Missed, and the Evidence a Composition Forecast Owes

Life of Research Weather foundation models fine-tuned on reanalysis now issue global air-quality forecasts that match or beat a chemical transport model on most verification targets. A July 2026 study of one of them, Aurora, reports that…

preprint

A Better Predictor of People Is Not a Better Synthetic Person: What Compute Scaling Laws for LLM Social Simulation Measure, Why Hyper-Rational and Flattened Simulators Mostly Measure Something Else, and the Deviation Test That Would Separate Knowing About People From Knowing the Answer

Life of Research Language models are proposed as stand-ins for survey respondents and experimental participants, and a 2026 study asked whether the current scaling paradigm will make them faithful enough. Across 85 base models trained…

preprint

An Uncorrelated Pair Is Not a Separated Source: What a Disentanglement Score Certifies About Aleatoric and Epistemic Uncertainty, Why Correlation, Proxy Tasks and Ground-Truth Targets Rank the Same Methods Differently, and the Decision That Has to Be Declared Before Any of Them Counts

Life of Research Separating a model's uncertainty into an aleatoric part, attributed to noise in the task, and an epistemic part, attributed to what the model has not learned, is now a standard goal, and the 2024-2026 literature has…

preprint

A Recovered Trigger Is Not a Verdict: What Trigger Recovery Establishes About Whether a Language Model Was Poisoned, Why a Search Result Depends on Its Candidate Set, Its Objective and Its Acceptance Rule, and What a Poisoning Screen Has to Report

Life of Research A defender handed an open-weight language model wants to know whether it was poisoned, and the most direct answer on offer is to recover the trigger. A July 2026 benchmark of roughly 800 backdoored models reports that…

preprint

A Hard-Task Gain Is Not a Change of Shape: When Language-Model Progress Is One Number, Why a Difficulty-Specific Gain Is Defined Only Relative to a Scale, a Difficulty Coordinate, an Overlap in Ability and a System Boundary, and What a Claim of Non-Scalar Progress Has to Carry

Life of Research Progress in language models is usually reported as one number: a time horizon, a latent ability on a stitched scale, or an average benchmark score. Whether one number is enough is now disputed. Cross-sectional analyses…

preprint

A Missing Slice Is a Missing Measurement: What Language-Level Audits of Safety Datasets Establish, Why the Multi-Turn Residual and Cross-Lingual Transfer Break the Step From Data Gaps to Model Behaviour, and the Contrasts That Would Price Native Data

Life of Research Model providers cite multilingual safety benchmarks as evidence that their models are safe for speakers of many languages, and a 2026 audit shows that such claims often dissolve when a single language is inspected: across…

preprint

A Residual Is Not a Third Factor: What the Gap Between a Perfect-Retention Ensemble and Joint Training Measures in Continual Learning, Why Chunk Size, Repetition and the Combiner Move It, and the Controls That Would Make Co-observation a Mechanism

Life of Research Continual learning is scored against two reference points: naive sequential training below and joint training on all the data above. The usual reading is that the distance between them is made of forgetting and lost…

preprint

A Rendered Passage Is Not a Picture: Four Things Called Modality Preference in Multimodal Knowledge Conflict, Why the Reported Reversal of Text Dominance Changes the Channel but Not the Content, and the Reliability-Matched Test That Would Settle It

Life of Research A recurring finding about multimodal language models is that they trust text over images when the two disagree. A 2026 study of thirteen models reports the reverse for one kind of evidence: a passage that contradicts the…

preprint

A Recall Score Is Not a Use Forecast: What Factual-Memory Benchmarks Predict About Whether a Conversational Memory System Uses What It Stores, Why the Published Transfer Failure Runs Through the Harness and the Write Policy, and the Within-Harness Test That Would Settle It

Life of Research Long-term memory systems for conversational assistants are compared almost entirely on question-answering benchmarks that ask for a fact stated somewhere in a long history. A 2026 benchmark for what its authors call memory…

preprint

A Checklist Is Not a Simplification: Where the Published Gains of Decomposed LLM Evaluation Come From, Why a Controlled Null on Holistic Criteria Does Not Reach Instance-Specific or Verifiable Decompositions, and the Ablations a Decomposition Claim Owes

Life of Research Automatic evaluation with a language model as judge is often improved, in published work, by splitting the judgment into parts: sub-criteria, checklist questions, rubric items or atomic claims, each scored separately and…

preprint

A Gap to a Reference Is Not a Loss: What Text-Capability Comparisons Between Vision-Language Models and Their Language Backbones Measure, Why Independent Audits and Vendor Reports Disagree in Sign, and the Controlled Trajectory That Would Assign the Cause

Life of Research A vision-language model is usually built by attaching an image encoder to a pretrained language model and training the combination on multimodal data, and a recurring claim is that this damages the language model's own…

preprint

A Detector Score Is Not a Release Risk: What PII Detection Benchmarks Certify About Deployment, Why 'Rivals Human Annotators' and 'Fails Under Shift' Are Both Measured, and the Coverage Audit a Release Decision Needs

Life of Research Automatic detection of personally identifiable information (PII) is the step that decides what text may leave an organisation, and its published record appears to contradict itself. Clinical de-identification systems have…

preprint

A Held-Out Site Is a Sample of One: What Random, Spatial and Site-Disjoint Splits Each Estimate, Why Leakage That Differs Between Configurations Reorders a Comparison, and How Many Sites a General Claim Needs

Life of Research A September 2026 study of farm-machinery traversability reports that a forward-facing near-infrared camera appears to beat a colour camera on a paddy-boundary class under its corpus's own sequence-level split, that the…

preprint

A Clean Answer Is Not a Clean Agent: What an Unlearning Certificate Covers Once the Model Calls Tools, Why the Agentic Leaks Divide Into Residue, Scope, Reacquisition and Exposure, and the Source-by-Observer Audit That Would Assign Each to Its Owner

Life of Research Unlearning benchmarks judge forgetting by questioning a bare model and reading its answer. A September 2026 benchmark argues that a certificate issued this way does not survive deployment: wrapped in a tool-using agent…

preprint

A Ceiling Score Is Not a Physical Inference: Why Pretrained Video Encoders Look Physically Competent on One Benchmark and Near Chance on Its Successor, What a Clean Scene Lets a Readout Read Off, and the Rankability Checks a Physics Evaluation Owes

Life of Research Pretrained video and image encoders are now chosen as the perception front end of world models for manipulation partly on the strength of evaluations that are said to measure physical understanding…

preprint

The Incumbent Did the Work: What the Simple-Baseline Results in LLM-Guided Program Search Settle, Why the Dispute Over Archives, Diversity and Test-Time Training Is Fought on Benchmarks That Cannot Rank It, and the Comparisons That Would

Life of Research Systems that let a language model write programs, score them with an exact verifier and feed the best ones back have improved published bounds on open mathematical problems, and each new system has added machinery: program…

preprint

Picking the Branch Is Not Solving the Problem: What an Imperfect Neural Operator Buys a Physics-Informed Network, Where Its Basin Information Comes From, and the Comparators That Decide Whether the Hybrid Pays

Life of Research Physics-informed neural networks (PINNs) can minimise a PDE residual and still land on the wrong solution, while neural operators are cheap to evaluate but lose accuracy outside their training range. A 2026 study joins the…

preprint

A Best Match Is Not a Twin: What Feature Similarity Between State Space Models and Transformers Certifies, Why the Null and the Shared Inputs Decide the Verdict, and the Controls That Would Show Two Architectures Learned the Same Thing

Life of Research Selective state space models such as Mamba compress the past into a fixed-size state, while Transformers attend over the whole context, and a growing set of studies asks whether the two learn the same internal features. A…

preprint

Which Fine-Tuning Failed? The Reported Failure and Recovery of Fine-Tuning in Model Editing Are Findings About the Edit Protocol, Why the Breadth-First Recovery Changes What the Editor May Remember, and the Matched Comparisons That Would Assign the Credit

Life of Research Fine-tuning has been treated as the weak baseline of model editing since the field's first benchmarks. A 2025 study argues that the weakness was an artefact of implementation: when edits are optimised in epochs over…

preprint

Calibrated Is Not Informative: What a Fall in Expected Calibration Error Shows About LLM Confidence, Why Verbalised Scores Fail on Resolution as Well as Level, and the Recalibration Baseline the Ensembling Results Omit

Life of Research A common finding in work on language-model confidence is that a model asked to state its own confidence in one pass is badly overconfident, and that some added procedure - sampling several answers, a separate critique…

preprint

Tiny in Parameters, Not in Compute: What a Recursive Reasoning Model's Puzzle Score Credits to Recursion, Why the Ablations Split by Task, and the Matched Comparisons That Would Assign the Rest

Life of Research Recursive reasoning models, which apply one small network many times to a latent state, report accuracies on Sudoku, maze and ARC-AGI puzzles above those of most frontier language models in their comparison tables, with 5…

preprint

A Starting Score Is Not a Ceiling: When Accuracy Before Reinforcement Learning Predicts Accuracy After It, Why Each Proposed Replacement Sees a Different Failure, and the Axis Test That Decides Which Proxy to Trust

Life of Research Reasoning models are usually built in two stages, supervised fine-tuning and then reinforcement learning against a verifier, and the first stage is usually judged by the benchmark accuracy of its checkpoint. An August 2026…

preprint

A Misalignment Rate Is Not a Model Property: Capacity, Propensity, Efficacy and Targeting in Multi-Agent LLM Commerce, Why a Null Capability Correlation Does Not Contradict the Scaling of Deceptive Skill, and the Design That Would Separate Them

Life of Research As language-model agents begin to trade with one another on behalf of separate principals, a natural worry is that the strongest agents will mislead, pressure or collude against the weakest. A 2026 study of 2,583 emails…

preprint

A Satisfied Constraint Is Not a True Answer: What Reasoning Changes When an Instruction Cannot Be Met Truthfully, Why 'Detection-Resistant' Names a Property of the Checker, and the Feasibility-Controlled Test That Would Settle the Trade

Life of Research A language model asked for something its knowledge cannot honestly supply has four ways out: it can break the instruction openly, decline, bend a real fact until it fits, or invent one. A January 2026 study reports that…

preprint

Which Loss Prices the English Share? What Target-Language Validation Loss Can and Cannot See in Continued Pretraining for Language Adaptation, Why Two Training Regimes Disagree About What English Buys, and the Instruments a Mixture Choice Needs

Life of Research When an English-centric language model is adapted to a new language by continued pretraining, practitioners usually keep a share of English in the training mixture, and a growing family of mixture laws chooses that share…

preprint

Optimism Needs a Way Back: What Value an Agent Should Give a State It Did Not Know Existed, Why Every Located Guarantee Borrows Its Answer From a Reset, a Data Horizon or a Reference, and What That Leaves for Unknown Unknowns

Life of Research A learning agent that reaches a state outside everything it has represented must give that state a value before it can act again, and the reinforcement learning literature offers three answers. Optimism values the unknown…

preprint

A Named Answer Is Not a Correction: The Moral Self-Correction Threshold Moved From 22B to 3.8B on Benchmarks a Context-Blind Policy Can Pass, and the Controls That Would Separate Steering From Correcting

Life of Research Instructing a language model to avoid stereotypes lowers its measured social bias, and a 2023 study reported that this capacity for "moral self-correction" emerges at 22 billion parameters. A 2024 study reported the same…

preprint

A Hidden Shift Is Not a Hidden Bias: What a Representation-Level Counterfactual Gap Certifies About an LLM Recommender, Why the Encoded-Versus-Expressed Studies Disagree, and the Placebo, Mediation and Transport Tests That Would Settle It

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Fairness audits of language-model recommenders compare the lists a model returns when a single stated identity is swapped and the rest of the prompt is held fixed. A…

preprint

A Rewrite That Keeps the Label Has Already Read It: Why an LLM Paraphrase Can Raise a Sentiment Classifier's Accuracy Without the Classifier Getting Better, Four Accounts of the Gain, and the Re-Annotation and Neutral-Rate Controls That Would Separate Them

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. A 2026 study reports that sentiment classifiers score higher on reviews rewritten by small language models than on the human originals, by up to 5.8 accuracy points…

preprint

Who Spoke Next Is Not Who Should Have: What Zero-Shot Failure on Multi-Party Turn-Taking Benchmarks Measures, Why the Human Reference Sits Beside the Models, and What Would Show That Knowing When to Speak Must Be Trained

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. An assistant that joins a group conversation has to decide, at every pause, whether to take the floor. Two 2026 benchmarks report that language models prompted…

preprint

A Blind Score Is Not a Reliance Measure: What Answering Without the Audio Shows About Audio-Language Benchmarks, Why the Partial-Input Debate Already Settled Half of It, and the Counterfactual-Audio Test the Other Half Needs

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Audio-language models are now routinely tested with the audio taken away, and the results are read in opposite directions. One 2026 audit reports that eight models…

preprint

Robust Because Coarse, or Robust Because Planned? A Research Agenda for Training-Data Provenance After the Aggregate-Disclosure Turn, in Which Granularity Buys Statistical Power, Prior Randomisation Buys Soundness, and Per-Example Value Has Neither

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Regulators have chosen coarse disclosure of training data: the template for the EU AI Act's public summary of training content asks for lists of large datasets, a…

preprint

Lossless at What Length? Near-Lossless KV Cache Compression Is a Claim Indexed by Decode Exposure, Scoring Statistic and Budget Accounting, the Decode-Aware Line Has Answered the First Index Mostly on Final-Answer Accuracy, and Divergence From the Full Cache Is Not by Itself the Defect

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Key-value cache compression is routinely reported as near-lossless at aggressive budgets: eviction, quantisation and merging methods keep a fifth or a tenth of the…

preprint

INT4 Is Not One Treatment: Why the Quantisation-Safety Literature Cannot Be Reconciled as Reported, How Six Design Axes Separate Its Studies Before Bit-Width Does, and Why Two 2026 Studies of the Same Model Names Reach Opposite Verdicts

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Post-training quantisation to 4 bits is widely treated as nearly free, a judgement resting on perplexity and multiple-choice accuracy. Whether it is also free for…

preprint

What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. A language-model agent is a model wrapped in a harness that builds its context, exposes its tools, decides when to retry and when to stop. Several 2026 studies set out…

preprint

One Pass, Three Purchases: Deduplication's Efficiency, Generalisation and Privacy Arguments Each Hold in Their Own Regime, Rest on Different Properties of the Repetition Distribution, and Cannot All Be Bought With One Similarity Threshold

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Deduplication became a fixed stage of language-model pretraining on the strength of three claims made together: removing duplicates saves compute, improves what the…

preprint

Deskilling Is Not One Endpoint: Five Quantities Behind Assisted Gains and Unassisted Losses in Clinical AI, Why Two Studies of the Same Design Disagree, and Why a Within-Clinician Trial's Control Arm Has Already Been Exposed

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Clinical AI tools are approved and bought on one endpoint, the performance of a clinician working with the tool, while a second literature asks what happens to the…

preprint

Dictionaries Without Ontologies: What a Sparse Autoencoder Latent Is Evidence For, Why Decodability, Use and Reproducibility Do Not Add Up to Canonicity, and the Identifiability Premise No Real-Model Result Has Checked

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Sparse autoencoders are now the default instrument for decomposing language model activations, and their latents are routinely reported as the model's features: units…

preprint

Three Hazards Under One Horizon: Constant-Hazard, Degrading-Step and Log-Logistic Accounts of Agent Failure Describe Different Levels of the Same Data, the 50 Percent Trend Barely Depends on Which Is Right, and Every High-Reliability Forecast Does

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Forecasts of autonomous AI capability increasingly rest on one number: the length of task, measured in human working time, that an agent completes half the time. That…

preprint

A Contamination Flag Is Not an Inflation Estimate: What Benchmark Contamination Detectors Certify, What Decontamination Is Meant to Buy, and Why the Clean-Subset Comparison That Joins Them Is Biased From Both Sides

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Almost every large language model report now includes a contamination analysis: benchmark items that overlap the training corpus are flagged, the model is rescored on…

preprint

Age Is Not Evidence of Staleness: A Recency Decay Is a One-Rate Survival Model, the Rate Belongs to the Fact Rather Than the Clock, and the Silent Change Only a Rate Could Catch Is the Case No Agent-Memory Benchmark Scores

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Language agents that keep memories across sessions usually discount a record by how long ago it was written or last read. That discount was introduced to decide what…

preprint

Did the Model Change, or Did Your Measurement? A Drift Claim Is a Test Against a Declared Null, the Careful Longitudinal Studies Mostly Caught Announced Version Moves, and a Capability Change Under a Pinned Identifier Is the Claim the Evidence Supports Least

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Two bodies of work describe the same deployed language models in opposite terms. One reports that models behind a fixed product name change over time and calls for…

preprint

Detectors Say 'Different', Not 'What': Score, Deviation and Name Are Three Claims With Three Referees, Only the Name Needs a Referee Outside the Detector, and Published Results Show Novelty Detected Far More Reliably Than It Is Named

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. An anomaly or novelty detector reports that an input is unlike the data it was built from. Users want more: they want to know what the unusual thing is. Explainable…

preprint

The Precursor Assumption: What an Early-Warning Threshold Must Deliver in a Frontier Safety Framework, Why the Continuity Evidence Covers Aggregates Under Fixed Elicitation Rather Than the Thresholded Task, and the Lead Time No Published Record Measures

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Frontier safety frameworks decide when a model needs stronger safeguards by testing it against capability thresholds at a fixed cadence, and they place an…

preprint

Spend the Budget on the Known or the Unknown? Open-Set Active Learning Scores Two Opposed Objectives Under One Name, Its Filtering Side's Own Results Do Not Treat Unknowns as Waste, and the Winner Is Set by a Relevance Label and a Query Price the Benchmarks Fix in Advance

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Active learning chooses which unlabelled examples a human should label. When the unlabelled pool contains classes the model has never been shown, two lines of work…

preprint

The Evaluator Is the Bottleneck: What a Self-Modifying System's Acceptance Test Must Satisfy Before Its Verdict Can Be Trusted, Why Co-Evolving Evaluators Shrink the External Anchor Rather Than Remove It, and How Moving the Anchor Outside Moves the Attack Surface With It

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Self-modifying AI systems now decide which changes to their own code to keep by running an acceptance test and retaining whatever scores higher. Published coding-agent…

preprint

Whose Goals? Autotelic Agents Generate Goals Within Spaces They Are Given: The Goal Space and the Referee Stay Outside the Agent, and a Foundation Model in the Loop Relocates Them Rather Than Removing Them

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Autotelic agents are described as learning to represent, generate, select and solve their own goals, and a new generation of systems built on foundation models is…

preprint

Problem Choice Without a Referee: An Automated Novelty Check Certifies an Empty Search, a Rediscovery Benchmark Credits the Hypothesis Humans Found Next, and Every Located Referee of Worth Independent of Proposer and Field Needs the Objective Given

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Pipelines that claim to automate scientific discovery gate their search on a judgement that a proposed idea or problem is novel and worth pursuing. The best-controlled…

preprint

Errors That Should Compound, and When They Do: The Product Rule Assumes Independent Steps, Conformal Guarantees Assume Exchangeable Instances, and the Unit of Independence Decides What Each Can Claim

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Two bodies of work quantify uncertainty in multi-step language-model reasoning, and both rest on an independence assumption placed at some unit. Step-level error…

preprint

Fingerprints That Survive What? Model Lineage Names Three Relations, Robustness Is Indexed by Which Party Is Adversarial, and a Benchmark's Distillation Column Scores the Parent That Was Not Distilled From

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Language-model fingerprinting is asked to answer questions of the form "is this model derived from that one?", and its methods are routinely reported as robust to…

preprint

Drift or New Class? Without Labels a Drifted Class and a New One Can Produce the Same Stream, the Two Lines of Work With the Most Explicit Assumptions Each Get an Answer by Freezing the Variable the Other Lets Move, and No Located Benchmark Scores the Attribution

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. A classifier deployed on a stream eventually sees inputs its model does not explain. Two different events can produce them: a known class can have drifted, or a class…

preprint

A Trust Score Needs a Consumer: Four Places a Per-Source Trust Value Can Act on an LLM Agent, the Missing Test of a Graded Discount Against an Adaptive Attacker, and Why the Defences That Report Guarantees Use a Gate Instead of a Score

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Work on trust between language-model agents produces two kinds of object. Protocol work produces identity, attestation, stake and constraint, all bound at the…

preprint

Self-Model or Self-Simulation? A Machine Self-Awareness Index Averages Sub-Scores With No Common Referent and No Fixed Sign, Why Persistent Identity, Goal Stability and Memory Continuity Are Not Evidence of Self-Access, and the Validity Tests Any Composite Would Have to Pass

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Some proposals to quantify machine self-awareness combine several sub-scores - persistent identity, goal stability, cross-session memory continuity, contradiction…

preprint

Four Things Called Forgetting: Interference, Transience, Reset and Unlearning Remove Different Things, Why Forgetting Looks Easy by Accident and Hard on Purpose Only Under Different Instruments, and the Savings Measurement That Would Tell Them Apart

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Machine learning uses one word for four operations. Catastrophic forgetting is damage that fine-tuning does to earlier capabilities. Transience is the fading of…

preprint

How Many Classes Are Out There? In Category Discovery the Class Count Is Granularity Carried Over From the Labelled Set, Supplying It Can Supply the Taxonomy, and Without It the Count Is Set by a Hyperparameter

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Category discovery asks a model to sort an unlabelled image collection into classes, some of which it has never been shown, using a labelled subset of other classes as…

preprint

The Allowance Is Doing the Work: Constructed Premises in Chain-of-Thought Step Verification, What Formal Validity Still Certifies Once They Are Permitted, and the Dependence Test Argumentation Runs and Step Verification Does Not

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. A verifier that checks the steps of a chain of thought cannot demand that each step state everything it relies on, because no real step does. Every published step…

preprint

Measured Against an Incomplete Answer Key: What Unknown Recall Certifies in Open-World Object Detection, Why Its Two Cost-Side Instruments Disagree, and the Separation That Was Published and Not Adopted

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Open-world object detection asks a detector to put boxes on objects whose classes it was never trained on, and the field's headline number for that ability is unknown…

preprint

Near-Zero Until Someone Tries: What a Prompt-Injection Defense Number Measures, Why Static and Adaptive Results Do Not Reconcile, and the Assumption the Out-of-Band Turn Has Not Yet Tested

Life of Research (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Several published prompt-injection defenses report attack success rates at or near one percent on static benchmarks; published adaptive attacks report success above…

About this research line

What has Pranay Mahendrakar published on Evaluation & Detection?

Pranay Mahendrakar has published 105 open-access papers on Evaluation & Detection: "A Query Update Is Not a Memory Write", "Calibrated Against Which Label? What Temperature Scaling Certifies When Annotators Disagree, Why a Majority Vote and an Annotator Distribution Each Make the Other's Calibrated Model Look Wrong, and the Target a Calibration Claim Has to Declare", "An Output Is Not a Memory Until Something Reads It Back", "Advice Is Not Effort", "A Verified Guess Is Not a Reversed Fact", "A Clause Boundary Is Not a Composition", "A Session Drop Is Not a Signal Ceiling", "A Fallback Is Not a Disposition", "A Pooled Accuracy Is Not a Scaling Verdict", "A Manuscript Score Is Not an Audit", "A Lower Score Is Not a Failed Transfer", "A Loop Count Is Not a Compute Dial", "A Disowned Turn Is Not a Detected Prefill", "A Coupled Explanation Is Not a Readout", "Told the Answer Is Not Taught to Reason", "A Right Response Is Not a Learned Law", "A Better Predictor of People Is Not a Better Synthetic Person", "An Uncorrelated Pair Is Not a Separated Source", "A Recovered Trigger Is Not a Verdict", "A Hard-Task Gain Is Not a Change of Shape", "A Missing Slice Is a Missing Measurement", "A Residual Is Not a Third Factor", "A Rendered Passage Is Not a Picture", "A Recall Score Is Not a Use Forecast", "A Checklist Is Not a Simplification", "A Gap to a Reference Is Not a Loss", "A Detector Score Is Not a Release Risk", "A Held-Out Site Is a Sample of One", "A Clean Answer Is Not a Clean Agent", "A Ceiling Score Is Not a Physical Inference", "The Incumbent Did the Work", "Picking the Branch Is Not Solving the Problem", "A Best Match Is Not a Twin", "Which Fine-Tuning Failed? The Reported Failure and Recovery of Fine-Tuning in Model Editing Are Findings About the Edit Protocol, Why the Breadth-First Recovery Changes What the Editor May Remember, and the Matched Comparisons That Would Assign the Credit", "Calibrated Is Not Informative", "Tiny in Parameters, Not in Compute", "A Starting Score Is Not a Ceiling", "A Misalignment Rate Is Not a Model Property", "A Satisfied Constraint Is Not a True Answer", "Which Loss Prices the English Share? What Target-Language Validation Loss Can and Cannot See in Continued Pretraining for Language Adaptation, Why Two Training Regimes Disagree About What English Buys, and the Instruments a Mixture Choice Needs", "Optimism Needs a Way Back", "A Named Answer Is Not a Correction", "A Hidden Shift Is Not a Hidden Bias", "A Rewrite That Keeps the Label Has Already Read It", "Who Spoke Next Is Not Who Should Have", "A Blind Score Is Not a Reliance Measure", "Robust Because Coarse, or Robust Because Planned? A Research Agenda for Training-Data Provenance After the Aggregate-Disclosure Turn, in Which Granularity Buys Statistical Power, Prior Randomisation Buys Soundness, and Per-Example Value Has Neither", "Lossless at What Length? Near-Lossless KV Cache Compression Is a Claim Indexed by Decode Exposure, Scoring Statistic and Budget Accounting, the Decode-Aware Line Has Answered the First Index Mostly on Final-Answer Accuracy, and Divergence From the Full Cache Is Not by Itself the Defect", "INT4 Is Not One Treatment", "What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable", "One Pass, Three Purchases", "Deskilling Is Not One Endpoint", "Dictionaries Without Ontologies", "Three Hazards Under One Horizon", "A Contamination Flag Is Not an Inflation Estimate", "Age Is Not Evidence of Staleness", "Did the Model Change, or Did Your Measurement? A Drift Claim Is a Test Against a Declared Null, the Careful Longitudinal Studies Mostly Caught Announced Version Moves, and a Capability Change Under a Pinned Identifier Is the Claim the Evidence Supports Least", "Detectors Say 'Different', Not 'What'", "The Precursor Assumption", "Spend the Budget on the Known or the Unknown? Open-Set Active Learning Scores Two Opposed Objectives Under One Name, Its Filtering Side's Own Results Do Not Treat Unknowns as Waste, and the Winner Is Set by a Relevance Label and a Query Price the Benchmarks Fix in Advance", "The Evaluator Is the Bottleneck", "Whose Goals? Autotelic Agents Generate Goals Within Spaces They Are Given", "Problem Choice Without a Referee", "Errors That Should Compound, and When They Do", "Fingerprints That Survive What? Model Lineage Names Three Relations, Robustness Is Indexed by Which Party Is Adversarial, and a Benchmark's Distillation Column Scores the Parent That Was Not Distilled From", "Drift or New Class? Without Labels a Drifted Class and a New One Can Produce the Same Stream, the Two Lines of Work With the Most Explicit Assumptions Each Get an Answer by Freezing the Variable the Other Lets Move, and No Located Benchmark Scores the Attribution", "A Trust Score Needs a Consumer", "Self-Model or Self-Simulation? A Machine Self-Awareness Index Averages Sub-Scores With No Common Referent and No Fixed Sign, Why Persistent Identity, Goal Stability and Memory Continuity Are Not Evidence of Self-Access, and the Validity Tests Any Composite Would Have to Pass", "Four Things Called Forgetting", "Refusal Is Not a Rate", "How Many Classes Are Out There? In Category Discovery the Class Count Is Granularity Carried Over From the Labelled Set, Supplying It Can Supply the Taxonomy, and Without It the Count Is Set by a Hyperparameter", "Does a Model Forget Differently When the Data Is Its Own? RL's Retention Advantage and Model Collapse Are Claims About the Same Loop, and No Study Has Measured Both", "Curriculum Is Three Claims, Not One", "Faithful to What? Four Instruments for Chain-of-Thought Faithfulness Disagree With Each Other, and the One Ground-Truth Check Run So Far Found Most of Them Near Chance", "When Deliberation Hurts", "Who Pulls the Plug? Self-Report, External Authority, and the Limits of Self-Isolation in AI Control", "Two Kinds of Missing", "Sufficient for Whose Answer? Three Predicates Under One Word in Selective Retrieval-Augmented Generation, and Why the Only One Tied to Correctness Cannot Be Computed Where the Gate Runs", "The Allowance Is Doing the Work", "Not Acting Is Not One Decision", "Measured Against an Incomplete Answer Key", "The Reassessment That Did Not Travel", "Attribution Is Scored on a Finished Trace", "A Failure to Reject Is Not a Finding", "Two Supply Chains, One Artifact", "Consistency Is Not Correctness", "Transfer Is a Directed Relation", "Predicting Your Own Failure Is Not Predicting Another Model's Advantage", "A Conflict Is Constructed Before It Is Measured", "An Unknown Unknown Is Defined Relative to an Oracle", "Near-Zero Until Someone Tries", "A Step Score Is Not a Step Verdict", "Ordering Is Not Resolution", "Out-of-Distribution With Respect to What? Four Referents Behind One Predicate, and Why OOD Detection Results Do Not Transfer to Foundation-Model Reasoning", "Automatic Benchmarks Measure an Asymmetry", "Intent Is Not a Property of the Record", "Poison Below the Base Rate", "The Crossover Does Not Carry the Claim", "Three Guarantees Under One Word", "A Second Model Is Not a Second Opinion", "The Self-Verification Gap", "Cross-Lingual Hallucination Patterns in Indic Languages", "AI-Generated Text Detection Under Paraphrasing", "Catastrophic Forgetting in Continual RLHF" and "The Emotional Intelligence Paradox in Large Language Models". Each is deposited on Zenodo with a permanent DOI and is free to read.

← All research topics