Reinforcement Learning with Verifiable Rewards (RLVR) has become a key paradigm to improve the reasoning capabilities of Multimodal Large Language Models (MLLMs). However, prevalent group-based algorithms such as GRPO require multi-rollout sampling for each prompt. While more efficient single-rollout variants have recently been explored in text-only settings, we find that they suffer from severe instability in multimodal contexts, often leading to training collapse. To address this training efficiency-stability trade-off, we introduce MSSR (Multimodal Stabilized Single-Rollout), a group-free RLVR framework that achieves both stable optimization and effective multimodal reasoning performance. MSSR achieves this via an entropy-based advantage-shaping mechanism that adaptively regularizes advantage magnitudes, preventing collapse and maintaining training stability. While such mechanisms have been used in group-based RLVR, we show that in the multimodal single-rollout setting they are not merely beneficial but essential for stability. In in-distribution evaluations, MSSR demonstrates superior training compute efficiency, achieving similar validation accuracy to the group-based baseline with half the training steps. When trained for the same number of steps, MSSR's performance surpasses the group-based baseline and shows consistent generalization improvements across five diverse reasoning-intensive benchmarks. Together, these results demonstrate that MSSR enables stable, compute-efficient, and effective RLVR for complex multimodal reasoning tasks.
Overview of the proposed MSSR approach. Given a multimodal input, i.e., an image and the corresponding question, we generate a single rollout through the policy model. We then use a Beta distribution to estimate the baseline value \( v \), compute the advantage \( A \), and normalize it across the batch. Finally, we propose entropy-based advantage shaping to preserve entropy and stabilize training.
Performance overview of MSSR. (a–b) Training and validation accuracy of MVSR (Multimodal Vanilla Single-Rollout), GRPO and our MSSR, trained on the Vision-R1-RL training set and validated on its corresponding validation set. MSSR remains stable and improves steadily, whereas MVSR is unstable and collapses. Notably, MSSR reaches a similar final validation accuracy to GRPO with half of the training steps, highlighting its superior training compute efficiency. (c) Our MSSR achieves higher generalization performance across diverse multimodal reasoning benchmarks, including MathVerse, MathVista, MMK12, R1-Onevision-Bench, and HallusionBench, compared to other baselines including GRPO, RLOO, and REINFORCE++. For fair comparisons, we have equivalent total number of rollouts per step for all methods.
| Model | MathVerse | MathVista | MMK12 | R1-Onevision Bench | HallusionBench | Avg. |
|---|---|---|---|---|---|---|
| SFT + RL | ||||||
| R1-Onevision-7B | 46.0 | 62.9 | 43.5 | 35.2 | 67.2 | 51.0 |
| OpenVLThinker-7B | 45.8 | 70.0 | 53.5 | 34.7 | 60.0 | 52.8 |
| VLAA-Thinker-7B | 48.2 | 68.0 | 51.7 | 38.4 | 70.0 | 55.3 |
| Zero RL | ||||||
| MM-Eureka-Qwen-7B | 50.3 | 71.2 | 61.7 | 39.1 | 66.4 | 57.7 |
| ThinkLite-VL-7B | 47.3 | 71.9 | 57.6 | 35.7 | 70.9 | 56.7 |
| Qwen2.5-VL-3B | 33.3 | 59.5 | 42.5 | 27.6 | 59.9 | 44.6 |
| + GRPO | 36.8 | 61.7 | 46.1 | 30.2 | 62.3 | 47.4 |
| + RLOO | 35.7 | 59.7 | 45.5 | 28.8 | 61.6 | 46.3 |
| + REINFORCE++ | 35.3 | 47.7 | 46.0 | 21.7 | 63.2 | 42.8 |
| + MSSR | 39.6 | 63.0 | 49.2 | 29.0 | 66.6 | 49.5 |
| Qwen2.5-VL-7B | 45.8 | 67.2 | 48.1 | 34.6 | 68.4 | 52.8 |
| + GRPO | 48.5 | 70.0 | 55.8 | 37.7 | 69.7 | 56.3 |
| + RLOO | 47.8 | 69.2 | 56.0 | 38.5 | 68.5 | 56.0 |
| + REINFORCE++ | 42.7 | 68.5 | 51.3 | 34.0 | 69.2 | 53.1 |
| + MSSR | 49.8 | 71.1 | 62.5 | 39.2 | 70.6 | 58.6 |
Ablation studies on effectiveness of techniques for preventing entropy collapse and stabilizing multimodal single-rollout training. Cross-modal regularization: This technique provides partial stabilization, increasing training accuracy but still resulting in degraded validation accuracy, and both metrics remain below those achieved by MSSR. Entropy loss: Adding an entropy loss term partially preserves entropy and improves training accuracy toward the end of training, but validation performance still degrades and entropy is not maintained as effectively as in MSSR.
Comparison of reasoning outputs from GRPO and MSSR. While GRPO produce incorrect answers, MSSR successfully solves the problem, demonstrating its superior reasoning capability. We highlight the critical reasoning steps that lead to GRPO's incorrect answer in red, and the key steps enabling MSSR’s correct prediction in green.
@article{liu2025stable,
title={Stable and Efficient Single-Rollout RL for Multimodal Reasoning},
author={Liu, Rui and Yu, Dian and Ke, Lei and Liu, Haolin and Zhou, Yujun and Liang, Zhenwen and Mi, Haitao and Tokekar, Pratap and Yu, Dong},
journal={arXiv preprint arXiv:2512.18215},
year={2025}
}