Most RLM writing is about where the numbers go up. This is the other list: the documented failure modes, and which of them you inherit the moment you build one.
The short version: RLMs fail in five reasonably predictable ways. The base model has to be a competent coder or the whole approach degrades. Recursion depth only pays off for strong models, and actively hurts weak ones. Telling a final answer apart from a passing thought is brittle. Cost is fine on the median and ugly in the tail. And without asynchronous sub-calls, the thing is slow. None of these are fatal. All of them are load-bearing if you are putting an RLM into production.
The original paper is unusually candid about this, which is worth saying, because the secondary coverage mostly is not.
Because that is the interface. An RLM does not read the long input; it treats the prompt as an environment and writes code in a REPL to slice, grep, sample, and dispatch over it. The root model's output is a program, not prose.
That has a hard consequence. As Zhang, Kraska, and Khattab put it, models without sufficient coding capabilities struggle as RLMs. In their evaluation, Qwen3-Coder trajectories contained significantly more syntax errors than GPT-5 trajectories, and each failed cell is a wasted turn.
The compounding version is worse. When a model that produces syntax errors also spawns sub-calls, the sub-calls inherit the same weakness. An error in a sub-RLM does not stay local; it returns junk to the parent, which then aggregates junk.
This is the most important practical filter on the whole approach. Our piece on small models and big contexts makes the optimistic case for 8B-scale RLMs, and it holds, but only for models post-trained for this specific loop. A generic small model is not a drop-in.
Sometimes, and the split is instructive.
With GPT-5 as the base, higher depth was the difference between good and best on the hardest task in the suite. On OOLONG-Pairs, a pairwise aggregation benchmark with quadratic complexity, the RLM scored 58% F1 at depth 1 and 76% at depth 3. On CodeQA, an RLM at depth 2 reached 66% against 58% for a compaction agent.
With Qwen3-Coder, depth greater than 1 showed diminishing or negative returns, for exactly the reason above: syntax errors that fail an output, propagated through sub-calls.
| Task | Base GPT-5 | RLM depth 1 | RLM deeper |
|---|---|---|---|
| OOLONG (131K tokens) | 44% | 56% | — |
| OOLONG-Pairs (32K tokens) | 0.1% F1 | 58% F1 | 76% F1 at depth 3 |
| CodeQA (23K–4.2M tokens) | 58% (compaction agent) | — | 66% at depth 2 |
So depth is a capability-dependent knob. If your base model is strong and your task requires genuine aggregation across the input rather than retrieval from it, depth buys you something real. Otherwise it buys you latency and failure surface. Note also what the 0.1% baseline means: on aggregation tasks, the long-context model was not slightly worse, it was non-functional. Retrieval benchmarks flatter it in a way aggregation benchmarks do not, which is the argument our post on context windows as the wrong abstraction makes at length.
The paper lists it plainly: distinguishing between a final answer and a thought is brittle.
In a single-pass model, the output is the answer by construction. In an RLM, the root model emits a stream of code, observations, intermediate notes, and sub-call results, and something has to decide which of those is the deliverable. Get it wrong and you either return a half-formed hypothesis as if it were settled, or you loop past a perfectly good answer.
Related, and equally unglamorous: thinking models without sufficient output tokens struggle as RLMs. A reasoning model that spends its budget deliberating has nothing left to emit the code that does the work. And the same system prompt across all models is problematic, because the loop each model needs is not the same loop.
Together those three are a warning about portability. An RLM harness tuned for one model is not a harness; it is a harness for that model.
Less than you would guess on the median, more than you would like at the tail.
On BrowseComp-Plus, a multi-hop benchmark over roughly a thousand documents totaling 6 to 11 million tokens, the RLM over GPT-5 at depth 1 averaged $0.99 per query, against an extrapolated $1.50 to $2.75 for the base model. Median RLM runs were cheaper than base model runs. Outlier trajectories, ones that recursed more than the task required, dragged the average upward.
That distribution shape matters more than the headline. A workload priced on the mean is priced on the outliers. If you are running an RLM at volume, the useful engineering targets are a recursion budget, a hard turn cap, and observability on the trajectories that blow past both. Our piece on inference-time compute covers the economics of paying for thinking rather than for parameters.
The paper's own phrasing is that RLMs without asynchronous language model calls are slow, which is a design defect stated as a finding.
The recursive structure is naturally parallel. A root model that decomposes an input into twelve independent sub-questions has twelve calls that do not depend on each other. Dispatch them sequentially and you pay twelve round trips end to end; dispatch them concurrently and you pay roughly one. The decompose-recurse-aggregate shape, which we cover in its own post, is defined by that independence at the recurse step.
So this limitation is real but self-inflicted. It belongs on the list because naive implementations hit it immediately, and because the sequential version is what most people build first.
Two categories are worth naming honestly.
Short inputs. If the whole thing fits comfortably in the window and the task is a single retrieval, you are adding orchestration, latency, and failure modes to buy nothing. Single-needle retrieval is close to constant-complexity work, and a plain call handles it.
Genuinely non-decomposable reasoning. If answering requires holding every part of the input in relation to every other part simultaneously, with no valid partition, the decomposition step has nowhere to go. Aggregation benchmarks like OOLONG-Pairs are near this boundary and RLMs still do well on them, which suggests the boundary sits further out than intuition says. But it exists.
The authors also flag that evaluation for harder, more natural long-context processing tasks remains highly under-explored, and that guardrail mechanisms still need development. Both are fair. The benchmark suite in the paper is a good suite; it is not the same thing as a production workload.
Not in the sense people usually mean. Every individual call inside an RLM is still a normal call into a normal context window, and it degrades on long inputs exactly the way it always did. The underlying curve is untouched.
What changes is that no single call is required to hold the whole input. The RLM stops climbing the degradation curve rather than flattening it. That is a real and large win, and it is a different claim from "long-context degradation is solved."
RLMs earn their results. A jump from 0.1% to 76% F1 on a quadratic aggregation task is not a benchmark artifact; it is the difference between a method that works and one that does not.
But the failure modes are specific and they are all upstream of the clever part. Pick a base model that writes correct code. Treat depth as a tuned parameter per model, not a dial that goes to eleven. Cap recursion and watch the tail of your cost distribution. Dispatch sub-calls concurrently. And accept that the harness you tune for one model does not transfer to the next one for free.
That is a shorter list than most new paradigms come with. It is not an empty one.