Meshy: A Role-Driven RL Training Framework under SPMD Paradigm

Tianyun Zhao$^{\dagger }$, Ao Sun$^{\dagger }$, Changlong Li, Yinghao Chen, Haoxuan Pan, Jinqian Zhang, Zekai Qu, Bingxiang He, ChaoJun Xiao, Xu Han

Github: https://github.com/OpenBMB/Meshy

$\dagger$ : Project lead

$^*$: Core contributors

<aside> ✨

We introduce Meshy, an open-source RL training framework without a central controller: Inference, Training, Rollout, and Teacher are independent services. Samples reach consumers directly through the TransferQueue data plane, and the workflow is driven by the readiness of data columns, bringing RL training back to the SPMD design used by pretraining.
Meshy gets rid of Ray and requires no RPC calls: one torchrun command starts all processes, and every process deterministically derives the topology locally from the same recipe, removing runtime task dispatch. Large tensors never pass through an orchestration process.
The same services can be freely composed through service-based role components to implement synchronous RL, bounded asynchronous training, fully asynchronous streaming training, and online policy distillation (OPD). Switching training modes only requires changing configuration fields; adding a role does not require modifying the framework core. A token-ring-based multi-role colocation mechanism lets arbitrary role combinations share the same GPUs.
Meshy is built on torchtitan and SGLang, with code open source on Github. </aside>

1. Introduction

Over the past few years, the workload of LLM RL systems has changed dramatically. Early RLHF focused on synchronous training: the model generated, scored, and trained in a fixed order, with clear synchronization boundaries between stages. Today, systems must also handle asynchronous generation, continual training, multi-turn agents, and interaction with external environments. States, versions, and failure boundaries that were implicit in synchronous execution now have to be managed explicitly.

The Single-Controller architecture, represented by verl[1], expresses heterogeneous distributed computation as a sequential program and effectively solves the most important orchestration problem of the synchronous RLHF era. Once execution is no longer strictly synchronous and tasks span multiple training rounds, however, Single-Controller gradually changes from an abstraction that simplifies the system into an obstacle to data transfer and task scheduling.

More and more systems are removing general-purpose distributed orchestration frameworks from the default dependencies of their core execution engines. A similar decoupling is happening in inference infrastructure: in its V1 engine, vLLM provides native execution paths for multi-node tensor and pipeline parallelism without Ray. vLLM processes are started independently on each node, and PyTorch torch.distributed establishes the cross-node process group.

Meshy emerged as a response to this changing landscape. We model roles like Inference, Training, Rollout, and Teacher as independent services instead of making a distributed orchestration framework the center of the RL system. All sample data flows between services through the unified TransferQueue[4] data plane. Data drives the control flow, while each process derives the topology locally from the same recipe on startup, bringing the RL workflow closer to the SPMD design commonly used by pretraining frameworks.

Meshy's structure naturally supports fully asynchronous training while remaining compatible with traditional synchronous training. Compared with a Single-Controller design, Meshy avoids various RPC calls and sheds the heavyweight distributed framework Ray, providing substantial advantages in debugging, performance, continuous integration, and code simplicity.

2. Background

2.1 The classic RLHF pipeline

Using PPO as an example, a round of traditional RLHF training looks roughly like this:

Prompts
   |
   v
Actor Rollout ──> Reference / Reward / Critic Forward
   |
   v
Advantage Estimation ──> Actor & Critic Update ──> Next Iteration

This pipeline has three important characteristics. First, synchronization between stages is explicit: rewards and advantages are computed only after the entire batch has finished rollout, and the next round uses the new weights only after the parameter update is complete. Second, the control flow of synchronous RLHF is fixed. Which stages run in each round and how they depend on one another are determined before training starts. Third, trajectories are generated in one shot. Most samples are the result of a single generation and do not require environment state to be maintained across rounds. Thus, although the computational components of traditional RLHF are heterogeneous (the Actor, Critic, and Reward Model may use different engines and parallelization strategies), its control flow is synchronous, regular, and predictable. In other words, the system is always in a determinate stage, and its state changes follow a globally unified execution timeline.

2.2 Single-Controller: expressing multi-node synchronous RL as a sequential program

Training and inference engines usually follow the SPMD paradigm: every process runs the same program and processes cooperate through collective communication. SPMD is well suited to expressing the forward and backward passes of a single model, but it is poor at expressing the top-level RL workflow. Rollout, Inference, and Trainer belong to different engines and use different parallel layouts, so an SPMD program is difficult to use for describing complex data flow and execution order. RL frameworks therefore usually add a control plane above the SPMD execution inside the engines to orchestrate roles.

Single-Controller offers a direct answer: if the system follows a single global execution timeline, write it as a sequential program. Google's Pathways[3] (arXiv:2203.12533) first presented this design systematically. verl's HybridFlow combines it with SPMD execution and has become a mainstream form for RL training frameworks: top-level control flow is concentrated in one process, and each role appears externally as a callable object:

for step in range(num_steps):
    sequences  = rollout.generate(prompts)     # one “function call”
    rewards    = reward.compute(sequences)
    advantages = estimate_advantage(sequences, rewards)
    actor.update(advantages)