Weekly Paper Notes — one of the top picks from the 2026-07-25 CS paper digest. Area: AI / ML (Agent Training).

Authors: Xiao Yu, Baolin Peng, Ruize Xu et al. arXiv: 2607.21557 · PDF

TL;DR

Modern agents are shaped less by their base model and more by the harness wrapped around it — Claude Code, Codex, OpenClaw, GUI-use scaffolds. But those harnesses are stateful, multi-process, and full of tool orchestration, which means existing open SFT/RL stacks (veRL, TRL, OpenRLHF) can’t natively express a rollout inside one. OpenForgeRL fixes this with two moving parts: (1) a lightweight proxy that intercepts every model call the harness makes, serves it via the training model, and quietly records the completion as a training sample; and (2) a Kubernetes orchestrator that runs each rollout in its own remote container so the harness’s real environment (browser, shell, container image, tool credentials) travels with it. The result is that any harness in any environment becomes an RL-trainable substrate. On tool-based agents, OpenForgeClaw hits 31.7 pass³ / 55.9 pass@3 on ClawEval and 33.7 on QwenClawBench; on GUI/browser agents, OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager — matching or beating models several times larger, using only hundreds to a few thousand training tasks.

Why this matters

The bitter lesson of 2026’s agent literature is that you cannot train an agent without training it inside the harness it will actually run in. Everything else — off-policy demonstrations, single-turn SFT, synthetic trajectories — leaves a distribution gap that shows up as brittle multi-step behavior at deployment. OpenForgeRL’s contribution is not a new algorithm; it is the plumbing that finally lets an open training stack sit behind a production-grade harness without either side needing to know about the other. The proxy-plus-orchestrator decomposition is the interesting part: the harness thinks it is calling a normal model API; the trainer thinks it is receiving normal rollout batches; the Kubernetes layer papers over the fact that every rollout wants its own filesystem, browser session, and toolchain. This is roughly the equivalent of the moment when RLHF stacks stopped shipping their own inference servers and started calling out to vLLM.

The paper’s most quotable empirical finding is on harness sensitivity: some harnesses (they call out ZeroClaw, OpenClaw, Codex) are meaningfully harder to learn against than others, and the choice of harness during training changes what the agent gets good at. RL improves self-verification, tool coverage, and multi-step plan completion — but error recovery remains a stubborn weak spot across the board, echoing what practitioners have been reporting anecdotally for a year. The negative result matters as much as the benchmark numbers.

Read alongside

  • ART / Openpipe (2025) — earlier attempt at agent-in-harness RL, single-machine scope.
  • veRL (2024) — the underlying RL codebase OpenForgeRL builds on.
  • OSWorld / WebVoyager / Online-Mind2Web — the GUI benchmarks used here; know the numbers before comparing.
  • AgentGym / Agent-R1 (2024) — cousins that took the “train inside the environment” bet a year earlier at smaller scale.

📄 arXiv abstract · 📄 PDF

Part of the Weekly CS Paper Digest series. Summary written from a close read of the preprint abstract; full-paper close read pending author code release.