Causal Representation Learning from Observational Video: Identifiability Assumptions and the Gap Between Synthetic and Real-World Settings

Abstract

Causal representation learning (CRL) — the recovery of latent causal variables and their causal graph from high-dimensional observations — has produced a substantial body of theoretical results in the past five years. CITRIS, CauCA, CRID, mechanismsparsity methods, and the recent grouping-based identifiability framework all establish conditions under which causal structure can in principle be recovered. The gap between this theoretical progress and what has actually been demonstrated on real-world observational video is wider than most accounts acknowledge. Almost all empirical CRL work uses synthetic 3Drendered video sequences with cleanly designed interventions, ground-truth latent factors, and assumption-friendly generative processes. Real video provides none of these. This paper argues that the question is not whether CRL is theoretically possible — it is, under specific assumptions — but which of those assumptions real observational video can plausibly satisfy and which it cannot. We make three claims. First, the identifiability literature implicitly assumes intervention access, multi-environment data, or strong sparsity priors that real video provides only partially and noisily. Second, three properties of real video — object near-independence, occlusions and scene changes as quasi-interventions, and naturally varying environmental contexts — offer plausible but unverified routes to satisfying identifiability assumptions. Third, objectcentric video models (Slot Attention, SAVi, SlotFormer) provide a necessary inductive bias but do not by themselves recover causal structure; the bridge between object-centric representation and causal identification is the genuine empirical frontier. We propose a research agenda focused on this bridge, on benchmarks that span synthetic-to-real, and on the methodological honesty needed to make cumulative progress.

About the author

Pranay Mahendrakar is an ai specialist and large language model engineer based in Bengaluru, India. He builds production artificial intelligence systems and publishes open-access research on how those systems fail. See all 70 papers by Pranay Mahendrakar, or his ORCID record.