RLM-Qwen3-8B was distilled from a larger model, but the same group has also trained RLMs with RL. Here is what each approach did, and what RL training an RLM requires.
The short version: Yes, and the original RLM paper already did a small version of it. RLM-Qwen3-8B was trained by distillation (rejection fine-tuning on filtered trajectories from Qwen3-Coder-480B-A35B), but the May 2026 revision of the same paper also reports a separate run that trained a Qwen3-4B-Instruct model as an RLM with reinforcement learning from verifiable rewards. That run generalized from a 32k-64k token task to a 512K-1M token one. The authors have since released an RL-trained adapter, RLM-Qwen3-30B-A3B, and the prime-rl harness that trained it.
By supervised fine-tuning, not RL. Appendix A of Recursive Language Models (Zhang, Kraska, Khattab, v3 dated 11 May 2026) says the authors ran Qwen3-Coder-480B-A35B-Instruct as an RLM root, with Qwen3-8B handling sub-calls, on 750 English LongBenchPro tasks. That produced 2,250 candidate trajectories. They removed runs that scored exactly 0.0 or never went past one turn, which left 1,072. The paper's main text rounds this to "1,000 filtered trajectories."
Each root turn then became its own training sample, turns past roughly 100k characters were dropped, and a script fixed templated mistakes such as misuse of FINAL and FINAL_VAR. Training used the prime-rl library with a batch size of 64 for 300 steps, about 48 H100 hours. The Figure 3 caption calls this "rejection fine-tuning" on distilled trajectories. The result beat the base Qwen3-8B by a median of 28.3% across the paper's four tasks.
The authors explain why they trained only the root: leaf sub-calls are "essentially general-purpose LLM requests," and "the major hurdle is learning to operate as the root model." We covered the data side of this, and why the trajectories were never released, in our post on public RLM trajectory datasets.
The same paper did, at small scale. Observation 6 in v3 describes training a Qwen3-4B-Instruct model as an RLM at depth 1 on MRCRv2, a synthetic task where the model must count and reproduce instances of a text in a corpus. The authors trained "purely through reinforcement learning with verifiable rewards (RLVR)" on the 32k-64k token split with 2 needles, for 150 steps, with a batch size of 128 and 4 rollouts per example. Each turn was capped at 4,096 output tokens and each rollout at 20 RLM iterations.
Every 50 steps they evaluated on the 512K-1M token split with 8 needles, and the paper reports that the RL-trained RLM generalizes to it. The exact scores appear only in a chart (Figure 3b), which also plots Gemini 3.1 Pro as a 1M-context reference, so we do not quote numbers here. The paper does not name the policy-gradient algorithm for this run. Note that the paper labels the model "Qwen3-4B-Instruct-0527," while the public Qwen3-4B instruct checkpoint is tagged 2507.
The second, larger example is the RLM-Qwen3-30B-A3B adapter from the same MIT group. Its model card says it was trained "using RL" with verifiable rewards and prime-rl, as a LoRA (rank 32, alpha 64) on Qwen3-30B-A3B-Instruct-2507, at depth 1, on 8 A100s. The card reports gains over the base model with the same prompt hint, for example 24.0 to 47.2 on OOLONG trec_coarse at 132k and 18.7 to 29.7 on BrowseComp-Plus. These are the authors' own numbers.
The code is in the training/ folder of the reference rlm repository: a verifiers-compatible environment for rlm.RLM at depth 1, a reward rubric, and an example config. Other public work has not trained RLMs. The reproduction study Think, But Don't Overthink (Daren Wang, March 2026) only evaluates existing models. Prime Intellect's January 2026 RLM post shipped an experimental RLMEnv but described RL training as future work.
The common starting point is GRPO from DeepSeekMath (Shao et al., 2024). GRPO drops PPO's value model and instead uses the average reward of a group of outputs sampled for the same question as the baseline. That matters for RLMs because a critic over a long, branching REPL transcript is expensive and hard to fit. The "4 rollouts per example" in the paper's MRCRv2 run is exactly this kind of group.
Multi-turn agent RL adds problems that single-answer math RL does not have. RAGEN (Wang et al., 2025) frames agent training as trajectory-level optimization (StarPO) and reports an "Echo Trap," where reward variance collapses and gradients spike. Their stabilized variant filters trajectories and adds a critic. They also report that without fine-grained, reasoning-aware rewards, useful reasoning does not reliably emerge, and agents fall back to shallow strategies.
The closest analogue to recursive sub-calls is Context-Folding (Sun et al., 2025). Its agent branches into a sub-trajectory for a subtask and folds it back to a summary, which is close to how an RLM hands a slice to a sub-call and keeps only the result. It trains this end to end with FoldGRPO, which adds process rewards for decomposition and context management, and reports an active context 10 times smaller than ReAct baselines.
Start with tasks that have checkable answers. That is why both the paper's RL run (MRCRv2) and Prime Intellect's environments (Oolong, math-python, verbatim-copy, DeepDive) use scored outputs instead of a judge model. Practical reward terms, roughly in order of how safe they are:
RLMTrainRubric) uses a single correctness function as its only reward.The rubric also logs iterations, REPL calls, sub-calls, and whether a final answer exists as metrics, not rewards. Watch them: a falling sub-call count with flat accuracy is an early sign of reward hacking.
Measure the trained model the way you would any RLM, on held-out lengths and task types. Our guide to evaluating an RLM covers that setup.
With outcome-only GRPO, you do not, at least not directly. DeepSeekMath's outcome supervision gives every token in an output the same normalized reward, so every root turn in a trajectory gets the same advantage. A good llm_query and a wasted one are pushed in the same direction if the final answer is right.
There are three ways to do better. You can freeze the sub-call model and train only the root, which is what the RLM paper did in both its SFT and RL runs, so sub-call quality becomes part of the environment. You can add process rewards, as DeepSeekMath's process supervision and FoldGRPO do, to score steps such as a well-scoped sub-call. Or you can treat each sub-call as its own episode with its own verifiable target, which only works when the sub-task has a checkable answer. The first is the simplest and the only one demonstrated on an RLM so far. In the reference harness, sub-calls go through a proxy to the trainer's inference server, so they are part of the environment, not tokens the trainer updates.
Much more than single-turn RL. One rollout is a full RLM trajectory: up to 20 root iterations in the paper's setup, each of which may fire many sub-calls, plus REPL execution time. Prime Intellect's environment uses a 120-second timeout per REPL call by default. Group sampling multiplies all of this by the group size.
The example config in the reference repo shows the scale of a modest run: 200 steps, a batch size of 32 with 2 rollouts per example, 4 GPUs for training and 4 for inference, and filters that drop zero-advantage and repetitive rollouts. The paper does not report compute for its RL run, only the 48 H100 hours for the SFT run. Two levers cut the bill: train on short splits and rely on length generalization, as the MRCRv2 run did, and run sub-calls in parallel, which the paper lists as future work. If you plan to try this, distill first: an SFT-warmed root gives RL a policy that already produces valid REPL code, so fewer rollouts are wasted on syntax errors. Base-model choice matters for the same reason, as covered in what base model you need for an RLM.
RL works for RLMs. The paper shows a Qwen3-4B RLM trained with verifiable rewards that generalizes from 64k to 1M-token inputs, and the released RLM-Qwen3-30B-A3B adapter is RL-trained. The headline model, RLM-Qwen3-8B, is a distillation result. For your own run, use GRPO-style group sampling, verifiable outcome rewards, a frozen sub-call model, and an SFT warm start. Treat credit assignment across sub-calls and rollout cost as the two unsolved parts.
No. It was trained by rejection fine-tuning on about 1,000 filtered trajectories from Qwen3-Coder-480B-A35B running as an RLM. Training used prime-rl with a batch size of 64 for 300 steps, about 48 H100 hours.
A Qwen3-4B-Instruct model trained as a depth-1 RLM with verifiable rewards on the 32k-64k token, 2-needle split of MRCRv2 generalized to the 512K-1M token, 8-needle split. The run used 150 steps, a batch size of 128, and 4 rollouts per example. The exact scores are shown only in the paper's Figure 3b chart.
Not to get started. The RLM paper trained only the root model and argues that sub-calls are essentially general-purpose LLM requests. Keeping the sub-call model frozen also makes credit assignment simpler.
Yes. The reference rlm repository has a training folder with a verifiers-compatible environment, a reward rubric, and an example prime-rl config for Qwen3-30B-A3B on an OOLONG environment. You still need to write an environment and correctness function for your own task.
Yes. The mit-oasys/rlm-qwen3-30b-a3b-v0.1 adapter on Hugging Face was trained with RL and verifiable rewards using prime-rl. It is a LoRA adapter, so it needs the Qwen3-30B-A3B-Instruct-2507 base model.
Usually yes. A root model that already writes valid REPL code wastes fewer expensive rollouts on syntax errors and empty answers. The RLM paper ran its SFT and RL experiments separately, so this order is untested on RLMs.