How Do You Run RLM-Qwen3-8B Locally with vLLM?

The short version: RLM-Qwen3-8B is real. Its weights are on Hugging Face as mit-oasys/rlm-qwen3-8b-v0.1, a full BF16 fine-tune of Qwen3-8B with an MIT license tag. Serve it with vllm serve, then point the rlms library at the local endpoint with backend="vllm" and a base_url. Two details decide whether it works: keep the context at about 32K tokens, and use the prompt format the model was trained on, which the current library no longer ships by default.

What exactly is RLM-Qwen3-8B, and where are the weights?

It is the small model the authors post-trained in the Recursive Language Models paper by Zhang, Kraska and Khattab. They fine-tuned Qwen3-8B on 1,000 filtered trajectories of Qwen3-Coder-480B-A35B acting as an RLM with Qwen3-8B sub-calls, on LongBenchPro tasks. The paper reports that it beats base Qwen3-8B as an RLM by a median of 28.3% across four tasks, and runs more than 3 times faster because it makes fewer mistakes.

The model card lists the name as RLM-Qwen3-8B-v0.1. It says the model "was trained on trajectories produced using a fixed system prompt" and "assumes the environment/scaffold from our RLM repo." It also recommends "using vLLM with our inference code" from alexzhang13/rlm. The repo holds four safetensors shards and a Qwen3ForCausalLM config. It is not a LoRA adapter, so you load it like any other 8B model.

How much GPU memory does it need?

Start with the numbers in the published files. The Hugging Face metadata counts 8,190,735,360 BF16 parameters. At 2 bytes each, the weights alone take about 16.4 GB. The config.json gives 36 layers, 8 key-value heads and a head dimension of 128. That works out to 147,456 bytes of KV cache per token in BF16, or about 4.8 GB for one full 32,768-token sequence.

So a 24 GB card holds the weights plus roughly one full-length sequence, with little room to spare. That matters for an RLM because llm_query_batched sends several sub-calls at once, and each live sequence needs its own cache. Our KV cache sizing guide walks through the formula. We have not benchmarked this model on specific cards, so treat these figures as a floor, not a fit test.

The vLLM memory guide lists the levers when you run out. Lower max_model_len to cap the context, lower max_num_seqs to cap batch size, use a quantized checkpoint, or split the model across GPUs with tensor parallelism. Its optimization page says vLLM pre-allocates KV cache from the gpu_memory_utilization fraction, so raising it gives more cache space.

How do you start the vLLM server?

vLLM's OpenAI-compatible server exposes /v1/chat/completions, which is the endpoint the RLM client calls. A minimal launch looks like this:

vllm serve mit-oasys/rlm-qwen3-8b-v0.1 --port 8000 --max-model-len 32768 --api-key token-abc123

Each flag appears in the vLLM docs. --port and --max-model-len are in the CLI reference, which also accepts 32K as shorthand for 32,768. --api-key is in the server guide, which warns that the key only guards the /v1, /v2 and /inference paths. Other endpoints stay open, so bind to localhost or put the server behind a proxy.

Add --tensor-parallel-size 2 if you split across two GPUs. If you want the base Qwen3-8B as a separate sub-call model, run a second vllm serve Qwen/Qwen3-8B on another port.

How do you point the RLM library at the local endpoint?

In the reference library, backend="vllm" routes to the same OpenAIClient used for OpenAI. The only difference is an assertion: base_url "is required to be set to local vLLM server address." The model name must match what vLLM serves, which by default is the Hugging Face path.

1. Install the library: pip install rlms.
2. Create the RLM with backend="vllm" and backend_kwargs={"model_name": "mit-oasys/rlm-qwen3-8b-v0.1", "base_url": "http://localhost:8000/v1", "api_key": "token-abc123"}.
3. Pass the key explicitly. The client only fills in an API key from environment variables for OpenAI, OpenRouter, Vercel and Prime Intellect URLs, so a local URL gets none.
4. Call rlm.completion(...) as usual.

Sub-calls go to the root backend unless you say otherwise. To send them to a second server, pass other_backends=["vllm"] and a matching other_backend_kwargs list. The constructor accepts exactly one extra backend. This mirrors the training setup, where the root was an RLM and the leaves were plain Qwen3-8B. Our post on what base model an RLM needs covers why a strong root with cheap leaves works.

Which system prompt and answer format does the model expect?

This is the step most likely to break a local run. The paper's appendix shows the prompt used for Qwen3-8B. It tells the model it has "a total context window of approximately ~32k tokens," that sub-LLMs share "this same ~32k token limit," and to aim for about 10K to 15K characters per llm_query call. The model learned to finish with FINAL(...) or FINAL_VAR(...). The authors note that 16% of raw teacher turns misused FINAL and 13% misused FINAL_VAR, so they patched those before training.

The library has moved on since then. A commit on May 13, 2026, titled "new dict-based final answer format," changed the default prompt. It now asks the model to set answer["content"] and answer["ready"] = True, and the main branch no longer mentions FINAL in its prompts or REPL code. The new default also tells the root that the sub-LLM window is about 500K characters, which is far too large for an 8B leaf.

You have two fixes. Pin a release from before the change: rlms 0.1.1 went to PyPI on February 18, 2026, and 0.1.2 followed on May 28. Or keep the current release and pass the paper's Qwen3-8B prompt through custom_system_prompt. Either way, check that the model's final answers actually end the loop before you trust any benchmark number.

What context length and sampling settings should you use?

Stay near 32K. The Qwen3-8B card gives a native context of 32,768 tokens and 131,072 with YaRN. The fine-tune's config keeps max_position_embeddings at 40,960 with no rope scaling. Qwen explains that 40,960 reserves 32,768 tokens for output and 8,192 for the prompt. It also advises against YaRN when contexts stay under 32,768, because static YaRN can hurt shorter texts. An RLM rarely needs more, because the long input lives in the REPL as a variable, not in the window. The authors also filtered out training turns past about 100K characters.

Set sampling values yourself. vLLM applies a model's generation_config.json by default. The base Qwen3-8B file sets temperature 0.6, top_p 0.95 and top_k 20, but the RLM fine-tune's file carries no sampling values at all. The paper says it used Qwen's recommended sampling parameters, and Qwen warns against greedy decoding for thinking mode. Pass them through sampling_args for the root and sub_sampling_args for the leaves. vLLM takes top_k through extra_body, and the RLM client merges an extra_body key from sampling_args. For split settings, see different temperatures for root and sub-calls.

The bottom line

Running RLM-Qwen3-8B locally takes one vllm serve command and a few lines of Python. Budget about 16.4 GB for the weights plus KV cache for each concurrent sub-call. Cap the context at 32,768. Give the client an explicit base_url and API key, and set Qwen's sampling values by hand. Most of all, match the scaffold the model learned: a 32K-aware prompt and the FINAL answer format. The model card says it "assumes the environment/scaffold from our RLM repo," and the scaffold has changed since the weights shipped.