Researchers have recently developed new approaches to enhance the reasoning reliability of large language models (LLMs) and proposed a benchmark system, Prism Benchmark OPG-D³, to better evaluate AI model generalization. The study provides new insights into how reinforcement learning strategies can improve LLM reasoning.

Led by Prof. DING Zenghui from the Hefei Institutes of Physical Science of the Chinese Academy of Sciences, in collaboration with researchers from other institutions, the related studies have been accepted by ACL 2026.

Current reinforcement learning approaches for LLMs mainly optimize models based on whether the final answer is correct. However, this answer-based approach may lead models to rely on superficial patterns instead of developing robust reasoning strategies.

Inspired by cognitive control mechanisms in the human brain, the researchers found that Process Reward Models (PRMs), which provide feedback on intermediate reasoning steps, can better guide models to refine their reasoning and reduce shortcut learning compared with Outcome Reward Models (ORMs), which only evaluate final results.

The researchers further found that simply increasing training data is not sufficient to overcome these limitations, while structured supervision of reasoning processes is important for improving model reliability.

They also introduced Prism Benchmark OPG-D³, an evaluation framework designed to test whether reinforcement learning models can generalize beyond familiar tasks. The framework includes the Oracle Performance Gap (OPG) metric, which measures the difference between a model trained on regular data and one optimized directly on test data.

Experiments on widely used benchmarks showed that current evaluation methods may underestimate generalization risks. The newly proposed Prism Benchmark OPG-D³ provides a more comprehensive framework for assessing model reliability under challenging and unfamiliar scenarios.

According to the researchers, the findings provide new insights into how LLMs learn and reason, highlighting the importance of improved training strategies and evaluation methods for developing more reliable AI systems.

Illustration of outcome reward-induced shortcuts and the corrective mechanism of process supervision. (Image by DING Zenghui)