Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL
2026-09-10 · Hugging Face
Overview
LoRA support recently landed in TRL's AsyncGRPOTrainer via PR #7017 and ships with TRL v1.14. The asynchronous trainer can now train an adapter instead of the full model, syncing only the LoRA adapter to vLLM. This post covers a real-world project where training and inference no longer share a machine.
Why LoRA Suits Reinforcement Learning
Thinking Machines's blog *LoRA Without Regret* demonstrates that LoRA can match full fine-tuning for policy-gradient RL, even with rank 1. The advantage function only provides ~O(1) bits of information per episode, so from a total-bits-of-information perspective, there is not much to learn from each step. A rank-1 adapter has enough capacity to absorb it.
Systems-Level Consequence
LoRA training also has a systems-level benefit:
- A rank-1 adapter for a 1.5B model is a few megabytes, while the full model is around 3 GB.
- After every update, only the adapter needs to be sent to inference workers instead of the full policy.
- vLLM can keep several adapters loaded at once; old rollouts finish with the policy they started with while new rollouts use the latest one.
The Challenge on Hugging Face Jobs
TRL's AsyncGRPOTrainer already separates training and generation. The trainer and vLLM can run on different machines at their own speed, which is straightforward in a single-node or cluster setting where processes share a filesystem or can form an NCCL group.
However, an HF Job is one container running on one VM. A single Job cannot spawn multiple nodes (limited to 8xH200 per node). AsyncGRPOTrainer is built for exactly this scale, so the key question became: how far can we get if we drop the requirement that the trainer and inference servers share a node?
Storage Bucket as Shared Filesystem
With full-weight sync, the answer would be "not far." Every update would need to move gigabytes between machines, which is what NCCL handles in dense clusters, but Jobs cannot communicate across nodes—no shared local disk, no shared localhost.
With LoRA, a sync is only a few megabytes. HF Jobs provide volumes backed by Storage Buckets, mountable as a FUSE filesystem in every Job, serving as a shared FS between nodes. No network path between Jobs is needed:
hf jobs run ... -v hf://buckets/aminediroHF/asyncgrpo-lora-buckets:/lora ...
Architecture
The final setup consists of:
- Trainer Job: runs AsyncGRPOTrainer with LoRA (and FSDP).
- Two vLLM Jobs: each serving the base model plus whatever adapter the trainer last published.
- Storage Bucket: mounted in all three Jobs at the same path, carrying the adapter from trainer to servers.
- Proxy server: routes each rollout to the replica most likely to hold its KV cache, and broadcasts adapter updates to all vLLM replicas.
Adapter Sync Path
The trainer does not send tensors to vLLM. Every few optimizer steps, it saves the adapter under `<output_dir>/.vllm_lora/trl-policy-v{N}`, publishes the directory with an atomic rename, then sends its path to vLLM's `/v1/load_lora_adapter` endpoint. vLLM loads files from disk, so the rollout worker can request `model="trl-policy-v{N}"`.
Under the hood, hf-mount exposes the bucket as a POSIX filesystem inside the container. Nothing in TRL or vLLM had to change—the trainer writes to `/lora/<run>/.vllm_lora/` and servers read from the same path.
Persistence and Recovery
Checkpoints and the final adapter are also stored in the bucket. HF Jobs are ephemeral, but a preempted trainer can resume training since the final adapter is always persisted to the bucket and never lost when the Job stops.
vLLM Replica Configuration
Each replica uses one GPU and the stock `vllm/vllm-openai` image, only needing to enable runtime LoRA loading and reserve enough adapter slots. The number of adapter slots follows from `max_staleness`: every weight sync bumps the policy version by one, and `max_staleness` is how many versions a rollout sample may lag behind the current policy before the trainer discards it.