A single accuracy number can make an RLM look revolutionary or pointless depending only on which benchmark you picked. This is how to build an evaluation that survives that objection.
The short version: a single accuracy number is close to meaningless for an RLM. What determines the result is the task's complexity class, the budget you gave the baseline, the recursion depth you happened to pick, and whether your grader can tell a final answer from a passing thought. An honest evaluation reports four things: accuracy broken out by complexity class, cost per query as a distribution rather than a mean, wall-clock latency, and a depth sweep. Everything below is the argument for why each of those is load-bearing.
This matters more for RLMs than for ordinary models, because an RLM is a harness plus a model plus a policy. Three things can move the number, and only one of them is the thing you meant to test.
Because long-context benchmarks are not one category. They differ in how much of the input a correct answer has to touch, and RLMs are sensitive to exactly that.
Consider two extremes from the original evaluation. S-NIAH is single needle-in-a-haystack retrieval: one fact, buried in filler, and the task is to find it. The work is constant-complexity regardless of input length. A long-context model handles it, and an RLM adds a code-writing loop, sub-calls, and latency to buy nothing.
OOLONG-Pairs is the opposite. It requires pairwise aggregation across the input, so the work grows quadratically with length. At 32K tokens, the base long-context model scored 0.1% F1. An RLM at depth 1 scored 58%, and at depth 3, 76%.
Those two results are from the same method. The 0.1% figure is the important one: on aggregation, the long-context baseline was not slightly worse, it was non-functional. If your evaluation suite is retrieval-heavy, you will conclude RLMs are an expensive wash. If it is aggregation-heavy, you will conclude they are transformative. Both conclusions are artifacts of the suite.
So the first requirement is a suite that spans the range on purpose, with the complexity class stated per benchmark. The original work used four: S-NIAH as the constant-complexity control, OOLONG and OOLONG-Pairs for semantic labeling and pairwise aggregation, CodeQA for repository-scale code understanding from 23K to 4.2M tokens, and BrowseComp-Plus for multi-hop question answering over roughly a thousand documents totaling 6 to 11 million tokens.
BrowseComp-Plus is worth singling out because of a design choice: it runs against a fixed, curated corpus rather than a live web search API. That separates retrieval failures from reasoning failures, which is precisely the confound that makes agentic benchmarks hard to reproduce. Our post on RLMs versus RAG covers why that separation is the interesting axis.
Four things, and three of them get skipped in most write-ups.
Not a suite average. An average over one retrieval benchmark and one aggregation benchmark describes no task that exists. Report per-benchmark, and label each with what the task requires: retrieval, filtering, aggregation, or multi-hop synthesis.
On BrowseComp-Plus, the RLM over GPT-5 at depth 1 averaged $0.99 per query against an extrapolated $1.50 to $2.75 for the base model. The headline is that the RLM was cheaper. The more useful detail is that median RLM runs were cheaper than median base model runs, and the mean was dragged upward by outlier trajectories that recursed further than the task required.
That distribution shape is the operational fact. A p50 and a p99 tell you whether you can run this at volume; a mean does not. Our piece on inference-time compute scaling covers the economics of paying for thinking rather than parameters.
Token counts are a poor proxy here, because the dominant latency factor in a naive RLM is whether sub-calls are dispatched concurrently. A root model that fans out twelve independent sub-questions and awaits them one at a time pays twelve round trips for work that is structurally parallel. Two implementations with identical token consumption can differ by an order of magnitude in wall-clock time.
Measure the thing users experience. If you report tokens instead of seconds, you are hiding your own implementation quality.
Depth is not a constant of the method. It is a tuned parameter whose optimum depends on the base model and the task, which makes reporting a single depth an incomplete result.
The original work used depth 1 for its main experiments, with the authors noting that the root model could call LMs but not other RLMs, and that depth 1 was sufficient for most modern long-context benchmarks. Where they pushed further it paid off on hard aggregation: 58% F1 at depth 1 rising to 76% at depth 3 on OOLONG-Pairs, and 66% at depth 2 on CodeQA against 58% for a compaction agent.
A subsequent reproduction study using open-source agentic models, DeepSeek v3.2 and Kimi K2, found the opposite direction on easier tasks. Deeper recursion caused the models to overthink: depth 2 degraded accuracy on simple retrieval and inflated execution time from 3.6 seconds to 344.5 seconds. That is not a rounding error, and it is invisible in any evaluation that reports accuracy at a single depth.
Four, because without them you cannot attribute the result to recursion at all.
Because the output is not a single object. A single-pass model emits an answer by construction. An RLM emits a stream of code, observations, intermediate notes, and sub-call results, and something has to decide which of those is the deliverable. The original paper names this directly: distinguishing between a final answer and a thought is brittle.
That is an evaluation problem before it is a product problem. If your grader takes the last emitted string, you will score a hypothesis as a final answer some fraction of the time, and that fraction differs by model. If your grader looks for a structured answer block, you will penalize models that failed to format rather than models that failed to reason.
The practical mitigations are unexciting and effective. Require a structured final-answer channel and count formatting failures separately from reasoning failures. Log the full trajectory so a disagreement can be adjudicated. And report the rate at which extraction failed at all, because a method that answers correctly but cannot say so is a different engineering problem from one that answers wrongly.
A related trap: thinking models without sufficient output token budget struggle as RLMs, because a model that spends its allowance deliberating has nothing left to emit the code that does the work. If you cap output tokens uniformly across models, you have introduced a confound.
A checklist that fits on one page:
Steps 6 through 8 are the ones that separate an evaluation you can operate against from a benchmark table. Our post on the decompose-recurse-aggregate pattern covers the structure those metrics are describing.
The authors are direct about this: evaluation for harder, more natural long-context processing tasks remains highly under-explored, and guardrail mechanisms still need development.
Both gaps are real. The existing suite is built from tasks that are easy to grade automatically, which selects for tasks with unambiguous answers. Much production long-context work, summarizing a case file, reconciling contradictory sources, drafting from a corpus, does not have one. Nobody has a good automatic grader for those, which means the benchmark numbers describe the easier half of the problem space.
The honest position is that current RLM evaluation demonstrates a large, replicated advantage on aggregation-heavy tasks with checkable answers, at competitive median cost, with a long tail. That is a strong claim. It is not the same claim as "RLMs are better at long context," and an evaluation that conflates the two is doing the reader a disservice. Our post on RLM limitations and failure modes covers the rest of that list.
If you read one thing from this: report the complexity class of every benchmark, run the depth-0 ablation, sweep depth, publish p99 cost and latency alongside the mean, and state the answer-extraction failure rate separately.
Five items. Most published RLM comparisons include one or two of them, which is why the numbers move so much between write-ups.