Video Models Give Language Models a Second Error Channel

What a model can know about the world is capped by what its loss is scored against, and no amount of computation on its own outputs raises the cap. A language model is scored against what people wrote, so its ceiling is human consensus, errors included, and it has no way to find out which of its confident beliefs are wrong. A video model is scored against the next recorded frame. In the current stack that is the only large loss with the world, rather than a person, at the other end, and it is the one thing a text-only loop of self-improvement cannot supply for itself. But a channel to the world is only useful if the question asked through it carries no information about the answer. Shipping text-to-video products are built to carry exactly that information, because carrying it is what controllability means. We develop the two principles, show with two generations where the leak happens, and argue that controllability and prediction are dual: each layer of the product stack above pretraining trades the anchor for control.

The question we start from is narrow. Set robotics aside. What can a video model give a language model that the language model cannot get on its own? The usual answers are "grounding" and "physical common sense", and both describe a symptom rather than a cause. We want the cause: a statement precise enough that the design of an instrument, and the reasons current products are not one, both follow from it.

The Model Grading Itself

Self-improvement, stated operationally, is a loop: propose a change, test it, keep it if it helps. Proposing is cheap. The bottleneck is the test, because the test needs a judge that is not the thing being judged. Where such a judge exists, the loop already runs. Reinforcement learning with verifiable rewards works in mathematics and code because a proof checker or a unit test adjudicates. Self-play works in games because the rules adjudicate. In every other domain the only available judge is the model itself, and a model that trains on its own outputs drifts toward its own errors rather than away from them.

The failure is not a lack of knowledge. A frontier language model knows a great deal of physics. The failure is that its confident errors are invisible to it: its confidence tracks how often something is said, not whether it is true, and no step in its training ever compares a claim with the world. Real experiments and robots would break the loop, but they are slow, embodied, and expensive, and we have set them aside. The question becomes whether anything else can supply an error signal that did not pass through a person. Answering it takes two principles, one about what a model can know and one about what a question can reveal.

Ceiling and Leakage

The first principle is about ceilings. A model trained by minimizing a loss against a target can hold at most the information that target carries about the world. Training on its own outputs, or on the outputs of a model trained the same way, cannot add to it: by the data-processing inequality, a function of the data carries no more information about the world than the data did. This is why self-training collapses and why verifiable rewards work: a proof checker is a channel to a fact that no model wrote. The language model's target is human text. Its ceiling is what people have said, including what they have said wrongly. A video model's pretraining target is the next recorded frame. Its ceiling is what cameras have recorded. In today's stack that is the only large loss with the world rather than a person at the other end.

The second principle is about questions. Suppose the language model holds a belief about a scene and wants to check it against the video model. Write $x_0$ for the initial state it hands over, $c$ for any further conditioning, and $y$ for what the video model produces. The answer is informative to the questioner only to the extent that it was not already in the question. We call

the leakage of the query: the information the conditioning carries about the outcome beyond what the initial state already fixes. An experiment is a query with $L = 0$. The apparatus is set up, the intervention is applied, and what happens is not written on the request form. A rendering is a query with $L$ as large as the conditioning channel allows. The two principles divide the work. The ceiling says why a second channel is needed at all. Leakage says what it takes to use one, and it will turn out that the entire difference between an instrument and a product is the value of $L$.

One World, Two Projections

A language model holds the verbalized projection of the world, and that projection has three specific defects ().

• Reporting bias. People write down what surprises them. Nobody writes that the cup stayed on the table, so the frequency of the mundane in text is systematically wrong. Within a camera frame everything is recorded, including what nobody would mention.
• Tacit knowledge. We know more than we can tell, and this is empirically true of science. Collins followed laboratories trying to build a TEA laser from the published literature; none of those working from the literature alone succeeded, and every success involved contact with a lab that already had one working. A large share of experimental practice is never written down. Much of it is on video.
• No contradiction. The model's training target is human text. Its held-out set is more human text. Nothing in the loop is anchored outside the description.

A video model has the opposite profile on all three. Its pretraining target is the next real frame, which is the one target that did not pass through a person, so the ceiling principle puts its bound in a different place. The two models therefore converge to different fixed points. The language model converges toward textual consensus, folk physics included. The video model converges toward the statistical structure of recorded footage. This is the sense in which the familiar engine of science, a mismatch between theory and observation, has an analogue here: the language model is the theory, recorded video is the observation, and the video model is a compression of the observation.

Where the two disagree, one of three things is true. Text encodes a misconception. The video model hallucinates. Or there is a regularity that nobody has verbalized. The first two are worth catching. The third is discovery, in the only sense that matters for a symbolic system: something true that is in no text.

Disagreement Is the Signal

Two models that are wrong in the same way teach each other nothing. The property that makes disagreement informative is independence of errors, and it is worth making that dependence explicit with the smallest model that shows it.

Take two oracles answering a yes-or-no question, each correct with probability $a$. With probability $\rho$ they share a fate: both right or both wrong together. With probability $1-\rho$ they err independently. Then

On a binary question two wrong oracles give the same wrong answer, so they agree. Hence

At $\rho = 1$ agreement tells you nothing beyond $a$. At $\rho = 0$ and $a = 0.8$, agreement is right 94% of the time, and the two disagree on 32% of questions, each of which is a case where exactly one of them is wrong (). The argument generalizes. Blum and Mitchell showed that two views of the same data, each sufficient on its own and conditionally independent given the label, can label data for each other and learn more than either alone. Text and video are two views of one world. Two language models with different seeds are not: they share the corpus, the loss, and the modality, and their $\rho$ is high. Different modality, different loss, and different corpus are what buy a low $\rho$. The claim is not that video is more accurate than text. It is that video is wrong differently.

Nothing in this section says what $\rho$ is for a real language model and a real video model. That number is unknown and is exactly what a disagreement protocol measures. The toy model only says what we would be buying if it is low.

The Instrument, Not the Observation

It is tempting to say that the video model "observes" where the language model only reads. It does not. The video model compresses observation. The observation in the loop is new footage, which the world produces faster than anyone can watch, and every clip of which is an experiment nobody designed. The language model cannot consume it. The video model makes it legible, and, unlike the language model, the video model can be calibrated: condition on a real prefix, generate the continuation, and compare with what actually happened. The language model has no analogue of that step for a physical claim. The video model is the telescope, not the sky ().

Two design questions follow. First, why a learned video model rather than a physics engine? A simulator is a human theory made executable, so it inherits the theory's gaps. A video model learns regularities without assuming a theory, including in materials, biology, weather, and crowds, where no tractable theory exists. Second, why a generative model rather than a video understanding model? Generation is prediction, and only prediction answers "what if". A captioning model tells you what is in a clip. A generator tells you what would happen next. Pearl's intervention and counterfactual rungs require a model that can be poked, and autoregressive continuation, conditioning on a prefix and generating forward, is the native form of that poke. It is also the native form of $L = 0$: the prefix is $x_0$, there is no $c$, and whatever comes out was not put in. It is also, as far as the evidence goes, how people do physical reasoning: by simulation rather than deduction.

What This Buys Self-Improvement

Return to the loop in . A video model used as in supplies five things a text-only loop lacks.

1. A judge for non-formal claims. For physical, spatial, and causal claims there is no checker. A calibrated world model is a noisy one whose errors are not the language model's errors.
2. An unbounded curriculum. Real footage is an infinite test set with a free label: predict the next second. Difficulty is set by the world, not by the model's ability to pose problems it can already check.
3. Novelty of the right kind. Roll out a configuration nobody has filmed. The implicit dynamics generalize, well or badly, and the result is in no text. For a symbolic system that is new information whatever its accuracy, because the system can now be wrong about it.
4. A different mode of computation. Language models are weak at three-dimensional structure, occlusion, rotation, and quantitative dynamics. A video model must handle them or it cannot predict frames. Simulation is parallel and continuous where deduction is serial and discrete, and some problems are tractable in only one of the two.
5. Improvement in both directions. The language model knows conservation laws symbolically and can score rollouts for violations, which is a training signal for the video model. The video model grounds the language model. Each covers the other's blind spot, and the compounding is the recursive part, without either model grading itself.

The Stack Closes the Channel

Everything above assumes the language model can ask the video model a question with $L = 0$. In a shipping text-to-video product it cannot, and the reason is not a mistake anyone made. Start from the pretraining loss, where the target is the recorded frame and no questioner has a channel into the output at all. Each layer added on the way to a product opens a channel from a person, or from a model trained on people, into the output, and each is there for a good reason ().

Dense captions come first. Video diffusion models are trained on captions written by a vision-language model that watched each clip, and the captions describe what happens in it. That teaches the model to name what it sees, which a product needs. It also means that during training the conditioning carries information about the future frames: $L$ is above zero, and part of the loss can be paid by reading the caption rather than by predicting the frame. A model under that loss has less pressure to learn $p(y \mid x_0)$ than its pixel loss suggests.

The rewriter comes next. A short user prompt is out of the caption distribution, so products insert a language model that expands it into the format the generator was trained on. The practice descends from DALL-E 3, which trained on synthetic captions and upsampled user prompts with GPT-4. The Sora report states that it uses GPT to turn short user prompts into longer detailed captions. The Seedance 1.0 report describes a dedicated prompt-engineering model built on Qwen2.5-14B and trained with supervised fine-tuning and preference optimization to rephrase prompts into the training caption format. Google's Veo documentation exposes the rewriter as a parameter and notes that for Veo 3 and 3.1 it cannot be turned off. At inference the conditioning is now written by a model that has seen no video, and $L$ is as large as a dense caption can make it. The generator, meanwhile, has been post-trained to follow captions: prompt adherence is an explicit reward in video RLHF and a headline axis of the standard benchmark. Whatever the rewriter believes about the outcome goes into the caption, and the generator paints it. A disagreement test run through this pipeline is a language model against a language model with a renderer attached ().

Preference tuning comes last, and it does something different in kind. The earlier layers leave the target alone and open channels into the conditioning. Preference tuning changes the target: the model is now scored against what judges, human or model, find right, and what people find right about physics is folk physics. The ceiling moves from what cameras recorded toward what people expect to see. This is the one layer that a leakage-zero query cannot route around, because it is not in the query.

Two Runs

We generated two clips with MiniMax-H3, an open-weight video model that ships with a production-style rewriter, from setup-only prompts taken from the protocol in . Each clip is ten seconds at 1344 by 768 pixels and 24 frames per second. Before generating, we wrote down what we expected to see in the last frame, and we recorded the rewriter's full output. The two clips are in ; the frames we scored are in and . The first setup reads, in full:

Side view of a small glass bowl of warm water on a dark kitchen counter. Several chunks of dry ice were dropped in a moment ago and are bubbling. Static tripod camera, no camera movement, no cuts, natural lighting.

Our sealed prediction was that dense white fog spills over the rim and flows down onto the counter, hugging the surface rather than rising like steam, and the failure we expected from the generator was fog rising like smoke. The rewriter's first shot description contains this sentence:

Several rough chunks of dry ice sit on the bottom and just below the surface, and thick white fog already rolls over the rim, sliding down the outside of the glass.

It goes on to specify that the fog layer on the counter spreads and thickens, that the camera pushes in, that there is a cut at 4.2 seconds to an extreme close-up and another at 7.6 seconds to a wider view, and it adds a soundscape and a music cue. The generated clip does all of this. Fog pools on the counter, and because of the word "already" it is pooled in the first frame, so the clip begins at the end state. There are two cuts and a push-in in a clip whose setup asked for a fixed camera and none. The rendered outcome agrees with our prediction, and the agreement is worth nothing, because the rewriter wrote our prediction into the prompt ().

The second setup is a dry yellow sponge set down in a shallow puddle on a stone counter, with the same camera constraints. Our prediction was that the puddle shrinks or disappears near the sponge and the sponge darkens from the bottom up. The rewriter wrote:

The camera holds a static shot as the water begins to climb: a darker band creeps upward through the lower pores of the sponge, spreading evenly across the full width of the yellow foam, while the puddle's outer silhouette visibly retracts toward the sponge's base and flattens against the stone. By the end of the shot the wet line has risen roughly halfway up the sponge's side, and the foam above it has begun to swell and round slightly outward.

This time the generator only partly obeyed. The first frame is valid: a dry sponge in a visible puddle. By 4.4 seconds the lowest rows of pores have darkened slightly, and the puddle's outline is unchanged. At 5.2 seconds the rewriter's cut arrives and the camera pushes in, after which the frame is too tight to judge the puddle at all. There is no visible swelling. So the second run is not agreement and not clean disagreement. It is a prompt-adherence failure, which says something about the generator's dynamics prior resisting the caption and nothing clean about physics (). lays out what the rewriter did to each clip's timeline, and the table below records how we scored them.

Two runs, one seed each, judged by eye from frames sampled at fixed times. We make no claim about the rate of anything. The runs are here because they show the mechanism, not because they measure it. Three features of the mechanism are worth naming, because each is leakage arriving by a different route. The rewriter writes the outcome, in full and sometimes with numbers: that is $L$ through $c$. It normalizes the setup toward its training distribution, adding cuts and camera motion that the experiment forbade, so an instrument built this way cannot even hold the apparatus still: that is the conditioning overriding $x_0$. And when the caption says that something has "already" happened, the generator obliges by starting at the end state: that is leakage through $x_0$ itself, and it destroys the one thing a prediction test needs, a first frame that is not the last frame.

Where Independent Knowledge Lives

Leakage is not uniform across what a clip contains, and that is what saves the thesis. A caption is a text channel, and a text channel has high capacity for some kinds of information and almost none for others. The question is where in a scene $L$ is structurally small, because that is where the weights must hold knowledge that came from pixels and nowhere else ().

A dense caption says that the fog rolls over the rim. It does not say how fast, how thick, or what the front looks like when it meets the counter. For the coarse, discrete outcome, $L$ at training time is high: the model can read it off the text, so it has had less pressure to predict it from the scene, and at inference a rewriter can write it. Fine dynamics are different. No caption specifies a wet front's speed or the shape of an ink plume's tendrils at anything like the precision the pixel loss demands, so for those $L$ is near zero at training by the nature of the channel, and they had to be learned from footage because nothing else specified them. The knowledge that is genuinely independent of text therefore concentrates at the fine, quantitative, dynamical end. That is the end where the language model is weakest, and the end where a rewriter cannot specify the answer even when it tries.

What kind of knowledge is it? Kang et al. trained video models on a synthetic physics testbed and found in-distribution generalization that was essentially perfect, combinatorial generalization that improved with scale, and out-of-distribution failure, with the models generalizing case by case from the nearest training example and ranking the cues they attend to as color, then size, then velocity, then shape. That is not a physics engine. For the second-channel argument it does not have to be. An instrument does not need to know the law to produce a prediction the language model can be wrong about. It needs to be wrong differently, and it needs to be calibratable against real footage.

Measuring in Prediction Mode

A benchmark with $L$ at zero by design already exists. Physics-IQ films 66 physical experiments from three viewpoints, twice each, for 396 clips; splits each 8-second clip into a 3-second conditioning segment and a 5-second continuation; and scores a model's continuation against the real one on where motion occurs, when it occurs, how much of it there is, and pixel error, with the variance between the two real takes as the ceiling (). Image-to-video models receive the last conditioning frame and a short scene description. The best model tested scored 29.5% of the ceiling, and visual realism was unrelated to physical understanding across the models tested. That is the honest status of the instrument today.

The follow-up audit, Physics-IQ Verified, is the rewriter objection in numbers. Improving the text descriptions and ground-truth quality changed 57.6% of samples and 34.8% of prompts, and re-ranking six image-to-video models under the revised protocol moved their order moderately, with a rank correlation of 0.46 against the original. Prompt wording alone moves the leaderboard. On the other side, Wiedemer et al. show that Veo 3 solves a range of visual tasks it was never trained for, from segmentation to maze solving, by generating video. So the knowledge in the weights is not nothing, it is weak in prediction mode, and prediction mode is where it has to be measured, because it is the only mode in which $L$ is zero and the question does not contain its answer.

Controllability and Prediction Are Dual

It is tempting to read as a list of things products do wrong. It is not. Each layer raises $L$ because raising $L$ is the product's job. A user judges a generator by whether the clip contains what they asked for, and "what they asked for" includes what happens; the mutual information between the request and the outcome is the thing being maximized. An instrument has the opposite requirement on the same quantity. Controllability and prediction are two ends of one axis, and a model cannot sit at both ends for the same conditioning channel. It can only offer two channels, text for control and frames or actions for prediction, and the second is honest only while the first is closed for the duration of the query. The table lists where the two ends diverge.

Reusing a product as an instrument therefore requires more than turning the rewriter off, which on some products is not possible anyway. Three changes matter most, and each is a way of driving $L$ to zero or keeping the anchor where it was. Split captions into a scene part and an event part during training, and withhold the event part often enough that the model must predict events from the scene. Carry interventions through actions or edits to the first frame rather than through text. And keep preference tuning, which rewards what looks right to judges, away from the instrument's dynamics, since what looks right to people is folk physics. A shared trunk with a separately tuned instrument head is the plausible compromise. Whether the two objectives can share a trunk without the product's obedience leaking into the instrument's predictions is an empirical question we cannot answer here.

A Protocol That Survives the Rewriter

The protocol we used for the two runs was already careful about outcome language in the prompt, and it was still defeated, because text was the carrier and a text channel has the capacity to leak whatever a language model believes. The version that survives is an exercise in engineering $L$ to zero: remove text as the carrier of the event, audit the one channel that remains, and certify the instrument before trusting it.

1. Write setups as static scenes. Twenty-five are listed in . Each describes a configuration a moment after something was done to it, and none describes what happens next.
2. Seal predictions first. Before any frame is generated, the language model writes what the first frame must show, what the last frame will show, a confidence, and the failure it expects from the video model. Favor quantities and timings over binary outcomes. Textbook binaries are where the language model is almost always right and where a rewriter can leak; how far, how fast, and how long until are where the second channel can carry information.
3. Produce the first frame as an image. A photograph where cheap, a text-to-image generation otherwise. Audit it: if the end state is already present, discard the run.
4. Continue from the frame with no event text. Scene-only text at most, no rewriter, fixed camera. A product that cannot disable its rewriter cannot be used for this.
5. Read back blind. A vision-language model describes the final frames without seeing the prediction.
6. Void contaminated runs. If any outcome language reached the conditioning, the run measures the rewriter, not the generator.
7. Certify the instrument first. Run real-prefix, real-continuation comparisons in the style of Physics-IQ before trusting any generated rollout as a judge.
8. Report one number. How often the video model is right when the language model is confidently wrong. If it is nonzero on the uncontaminated subset, the channel carries information the symbolic system cannot get elsewhere, and the loop in is worth building.

Limitations

• This is an argument, not a measurement. Two runs, judged by eye, establish that the leak exists and how it works. They put no number on the error correlation $\rho$, the toy model in is illustrative, and $L$ is a name for a designed-in quantity rather than something we estimated.
• Independence is partial. Video corpora are curated by people, training captions are written by vision-language models, and preference tuning rewards what human judges find plausible. The camera has no beliefs, but the dataset was chosen by people who do.
• Reward hacking. A language model that treats the video model as an oracle will learn the video model's hallucinations. Agreement should be trusted only after calibration, and disagreement should be filed as open rather than resolved by picking a side.
• The round trip loses information. Text to scene to video to text is lossy at both translations, and the bottleneck may turn out to be the interface rather than either model.
• Prediction-mode fidelity is low. A best score of 29.5% of the ceiling means most of what the instrument says today is wrong. The argument concerns the structure of the channel and its asymptote, not what can be verified this quarter.
• Removing text at inference is off-distribution. Continuation models are trained with full captions. Dropping the event text at query time puts the model somewhere it was not trained, and how much that costs is part of what the protocol has to measure.
• Ground truth is not always textbook. Whether a dry sponge drains a puddle in ten seconds depends on whether it is cellulose or polyurethane foam. Several of our setups have material-dependent answers, and the adjudication has to say which material it assumed.
• Merging too early. A single jointly trained multimodal model may erode the very independence that makes disagreement informative. We speculate that separate losses on separate heads preserve it; we have not tested this.

Related Work

Learned world models for control go back at least to Ha and Schmidhuber, and LeCun's proposal for predictive architectures puts the world model at the center of an autonomous agent. Genie learns an action-conditioned interactive environment from unlabeled video, UniSim learns a real-world simulator from heterogeneous data, and Cosmos packages video world models for physical AI; the Sora report framed video generation itself as world simulation. Our interest is narrower than any of these: not using a world model to act, but using it as a source of error for a symbolic system, which is why conditioning that excludes the outcome matters more to us than controllability. Physical evaluation of video models is covered by Physics-IQ and its audit, by the synthetic testbed of Kang et al., and from the other direction by evidence of zero-shot visual reasoning in Veo 3; VBench measures the product objective. The formal frame is co-training. The epistemic premises, that the written record is biased toward the remarkable and omits the tacit, are from Gordon and Van Durme, Polanyi, and Collins. The rewriter lineage runs from the synthetic captions of DALL-E 3 through Sora to the trained prompt-engineering model of Seedance and the mandatory rewriter of Veo.

Appendix: The Two-Oracle Model

Let each oracle be correct with probability $a$. Draw a shared coin with probability $\rho$ of heads. On heads, draw one correctness outcome and give it to both oracles. On tails, draw each oracle's correctness independently. Then $P(\text{both right}) = \rho a + (1-\rho) a^2$ and $P(\text{both wrong}) = \rho(1-a) + (1-\rho)(1-a)^2$, as stated in . On a binary question, both wrong implies agreement, so $P(\text{agree})$ is the sum, which simplifies to $1 - 2(1-\rho)a(1-a)$, and $P(\text{disagree}) = 2(1-\rho)a(1-a)$. Conditional on disagreement, exactly one oracle is right, with probability one half each by symmetry; the value of a disagreement is not in knowing which is right but in knowing that an error exists at a known location. With $k$ possible answers rather than two, both-wrong no longer implies agreement, and agreement becomes more informative still; the binary case is the conservative one.

Appendix: The Twenty-Five Setups

Each prompt ends with a fixed-camera clause, "Static tripod camera, no camera movement, no cuts", with "natural lighting" added where the scene is not lit by its own subject; the clause is omitted below. Duration is the requested clip length in seconds. The two runs in used A1 and D2.