4 / papers
How models hold context, carry state, and compose steps beyond next-token prediction.
State-space models did not merely "open a door" beyond attention; they walked through it. Mamba, Mamba-2, Samba, Jamba, Granite 4, Hymba, and a growing family of hybrid attention-SSM architectures are now in production language models…
Causal representation learning (CRL) — the recovery of latent causal variables and their causal graph from high-dimensional observations — has produced a substantial body of theoretical results in the past five years. CITRIS, CauCA, CRID…
Modern reasoning systems are dominated by autoregressive models trained with reinforcement learning on reasoning trajectories — the o1, R1, and Claude 3.7 family of "thinking" models. Energy-based models (EBMs) offer a theoretically…
In-context learning (ICL) — the ability of a transformer language model to acquire a new input-output mapping from a handful of demonstrations in its prompt, with no weight updates — remains the most striking and least understood…
