Optimism Needs a Way Back: What Value an Agent Should Give a State It Did Not Know Existed, Why Every Located Guarantee Borrows Its Answer From a Reset, a Data Horizon or a Reference, and What That Leaves for Unknown Unknowns

Abstract

Life of Research

A learning agent that reaches a state outside everything it has represented must give that state a value before it can act again, and the reinforcement learning literature offers three answers. Optimism values the unknown high so that the agent goes to look; pessimism values it low so that the agent stays where its knowledge holds; a 2025 proposal for agents with growing awareness values it at the average of what the agent already knows, on the grounds that nothing is known about it. In this analysis the author reviews and synthesises published results to show that none of the three rules is justified by the agent's ignorance, and that each located guarantee borrows its justification from something outside the rule. The located efficiency results for optimism assume the agent can always get back, through episodic resets or a communicating environment; where some states cannot be reached, optimistic algorithms incur linear regret, and where some states cannot be left, a 2020 result shows that any agent guaranteed to be asymptotically optimal is, under a stated condition on the environment, destroyed or incapacitated with probability one. Pessimism's minimax guarantees assume data collection has ended; when learning resumes, a value floor set too low makes a pessimistic agent discard a good initial policy, and on one benchmark suite a calibrated floor cut average cumulative regret from 0.41 to 0.22. The averaging rule's regret bound holds under an explicit assumption that unknown states are worth no more than the known average, plus resets. The paper consolidates these results in two tables and a figure, states the choice as a decision procedure whose first step is detecting that a state is new at all, and names the comparisons that would settle what an agent without resets, mentor or ceiling should assume.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 103 papers by Pranay Mahendrakar, or his ORCID record.