Verifier-Induced Support Reshaping in On-Policy Optimization

Shaohang Wei1, Zikun Su2, Feifan Song1, Wen Luo1, Wei Li1, Guangyue Peng1, and Houfeng Wang1

Peking University

1Peking University
2BUPT

arXiv preprint, 2026.

On-policy verifiers do more than score sampled trajectories. They reshape which behaviors remain reachable, rewardable, and trainable next.

Paper excerpts live at the page edge. Click one to scatter—and click again to recompose—the text.

Conceptual map of verifier-induced support reshaping Many possible trajectory routes approach a finite-rollout-budget verifier. A smaller, reranked set remains reachable on the other side. V possible trajectoriesverifierfinite rollout budgetreachable nexteffective support depends on budget B
Conceptual map. The verifier reranks routes within a finite sampling budget. This illustration is not experimental data.
Figure 1Paper overview
Paper overview contrasting backward-looking catastrophic forgetting with forward-looking support reshaping, then showing the bidirectional Math-RLVR and IF-RLVR study. Open figure
Figure 1. Catastrophic forgetting looks back at retained performance; support reshaping looks ahead at future trainability. The study tests both training orders with Math-RLVR and IF-RLVR.

Looking forward, not only looking back

A capability can remain possible in principle yet become too rare to appear within the rollouts available to the next on-policy stage.

Catastrophic forgetting

Asks what performance or previously learned capability remains after adaptation. It is retrospective.

Support reshaping

Asks what reward-positive trajectory remains likely enough to discover and reinforce next. It is prospective and budget-dependent.

Hand-drawn editorial illustration of two paper routes, one fading and one remaining open through a small gate.
Editorial interlude. Effective support asks what remains reachable within a finite sampling budget. Conceptual artwork—not experimental data.
“A verifier does more than improve what the policy does now; it changes what the policy can still learn next.”

Four measurements, one sampling question

Repeated verifier-scored rollouts reveal both average success and the boundary of what remains searchable.

pass@1 / rollout mean
The expected success rate of one stochastic rollout.
best@k
Whether at least one of k sampled responses succeeds.
pass-count
The number of successful rollouts for a prompt.
Joint support, J = C ∧ F
Correctness and strict following in the same response.

A bidirectional test of future trainability

Starting from the same policy in each model family, one branch receives Math-RLVR and is evaluated for instruction-following support; the other receives IF-RLVR and is evaluated for math support. Each branch is then continued with the opposite verifier.

Each training configuration has one fixed-seed run. Repeated rollouts measure sampling variation within a policy, not uncertainty across independent training seeds.

Shared BaseQwen3-8B-Base
Qwen2.5-Math-7B
Math-RLVRMeasure IF support
then continue with IF
IF-RLVRMeasure math support
then continue with math

Finding A

Average success increases while coverage under repeated sampling decreases

Math-RLVR moves mixed instruction-following prompts toward both extremes. An average rollout succeeds more often, yet a 32-rollout search finds a successful response for fewer prompts.

Δ pass@1+0.065average success
Δ best@32-0.098prompt coverage
Figure 2Polarization
Figure 2 shows pass@1 increasing, best@32 decreasing, and mixed instruction-following prompts shifting toward all-correct and all-wrong buckets under Math-RLVR. Open figure
Figure 2. Math-RLVR raises average IF success while polarizing prompt outcomes. Mixed prompts move toward all-correct and all-wrong buckets.
Final checkpoint minus Base
ModelBenchmarkΔ pass@1Δ best@32
Qwen3-8B-BaseIFEval+0.065-0.098
Qwen3-8B-BaseIFBench+0.032-0.067
Qwen2.5-Math-7BIFEval+0.079-0.114
Qwen2.5-Math-7BIFBench+0.016-0.037

Finding B

IF-RLVR lowers math searchability

AIME best@k falls at every tested budget, while visible openings move from deliberative-reasoning initiation (DRI) toward direct-answer initiation (DAI).

Pearson r = -0.85 is correlational. DRI and DAI classify the visible response opening; they do not reveal a hidden chain of thought or internal cognitive state.

Figure 3Correlation
Figure 3 shows AIME best@k declining for k equal to 4, 8, 16, and 32, DAI openings becoming dominant, and a negative checkpoint-level association between DAI share and best@32. Open figure
Figure 3. The result is behavioral and correlational: it does not show that mathematical ability is erased or that DAI alone causes lower searchability.

The response opening is a localized control point

Distributional analysis first locates the largest change. Controlled decoding interventions then test whether route selection itself changes downstream searchability. These are distinct pieces of evidence.

Hand-drawn editorial illustration of a small doorway, two paper trails, and a red route tile placed at the threshold.
Editorial interlude. A route-entry metaphor for the localized opening effect. Conceptual artwork—not a model-state diagram.

Localization

The first token moves most

Mean Jensen-Shannon divergence is largest at the first generated token in every tested model, RLVR branch, and benchmark combination. On AIME, the first-to-interior ratio ranges from 9.8× to 106.7×.

High-divergence positions mostly reuse candidates already supported by Base, although IF-RLVR can also promote an initially low-probability opening token.

Figure 6Localization
Figure 6 plots next-token Jensen-Shannon divergence by response position, with the largest average divergence at the first generated token across all tested settings. Open figure
Figure 6. Position-wise divergence localizes the largest policy shift but does not by itself establish a causal route effect.
distribution shift correlation locates intervention tests route entry

Controlled intervention

Changing route entry changes math searchability

From IF-RLVR checkpoints, forcing Base-side or DRI openings raises best@32 for both model families on AIME and MATH-500. A position sweep shows that the route-label effect is concentrated near the response opening.

The reverse intervention is not symmetric across model families. The evidence supports a causal, localized route-entry effect in the tested settings, not a broad rewrite or erasure of downstream reasoning.

Figure 8aRoute intervention
Figure 8a compares math best@32 under free decoding, forced route tokens, and forced DRI or DAI prefixes for Base and IF-RLVR checkpoints across two model families. Open figure
Figure 8a. Free decoding compared with forced route-token and prefix conditions.
Figure 8bPosition sweep
Figure 8b sweeps the forced IF-side route token across response positions, showing that DRI suppression occurs near the opening and recovers at later positions. Open figure
Figure 8b. The same IF-side token is forced at different decoded positions.
Hand-drawn editorial illustration of two torn paper fragments joined only partly by green thread.
Editorial interlude. A stitched-paper metaphor for partial, teacher-dependent preservation. Conceptual artwork—not experimental evidence.

Preservation levers help partially, not uniformly

Sequential training, reference-policy constraints, routing priors, and on-policy distillation each expose a different limit. None is a general solution in the reported experiments.

01

Sequence matters

After IF-first training, the mixed-support group falls from 33.6% to 1.6% by step 20 of Math-RLVR, leaving little within-group reward variation. This path does not prove irreversible loss.

02

Reference KL trades objectives

During Math→IF training, stronger reference constraints retain more math performance but yield smaller IF gains. No tested coefficient retains math while matching unconstrained IF adaptation.

03

Teacher state matters

With a converged IF teacher, MATH-500-128 mean@16 falls from 0.3433 to 0.0879 and shortcut responses rise from 4.5% to 18.8%. Across the sampled teachers, teacher and student shifts are negatively associated (Spearman ρ = -0.594); a coarse scan of four earlier states favors different target and retention outcomes.

Figure 9Routing prior
Figure 9 shows that a soft DRI-biased prior delays the shift toward DAI openings, but every tested condition reaches a DAI rate of one by training step 100. Open figure
Figure 9. A one-time DRI prior changes when the observed shift occurs, but not the endpoint route in this run.
Figure 11Teacher state
Figure 11 compares four IF-RLVR teacher checkpoints for on-policy distillation, showing different trade-offs between matched IFEval transfer, MATH-500 retention, and contentful versus shortcut responses. Open figure
Figure 11. Among four sampled teacher states, T2/40 gives the highest matched IFEval mean@16, while T1/20 is the only student that improves MATH-500 mean@16 (+0.93 pp). This scan does not establish a global optimum.

Marginal gains combine only partly in one response

MathIF and ReasonIF provide an independent endpoint stress test. Correctness C and strict instruction following F can improve in different responses; joint support requires J = C ∧ F in the same rollout.

Table 3. Marginal and joint support, averaged over 16 rollouts on 290 prompts per benchmark
BenchmarkCheckpointCorrectness CFollowing FJoint J = C ∧ F
MathIFBase0.3590.2500.094
Math-RLVR0.4720.2530.118
IF-RLVR0.3770.4310.148
ReasonIFBase0.4670.2390.109
Math-RLVR0.5220.2350.126
IF-RLVR0.4380.3910.171

Target-aligned marginal gains are substantially larger than the resulting joint gains. Improving one verifier does not automatically produce responses that are both correct and constraint-following.

Verifier choice is a forward-looking training intervention

Endpoint performance is not enough to characterize an on-policy training stage. Searchability and reward variation determine what a later verifier can still discover and amplify.

Effective rewardable support should be monitored across rollout budgets and plausible future objectives. Preservation methods should be evaluated jointly by target gain, retained cross-verifier support, and response quality—not by the current verifier alone.

Paper

Verifier-Induced Support Reshaping in On-Policy Optimization

arXiv:2608.00220 [cs.LG], with cs.CL as a secondary category. First submitted July 31, 2026.

Abstract PDF DOI

Correspondence: Houfeng Wang and Shaohang Wei.

Citation

@misc{wei2026verifier,
  title         = {Verifier-Induced Support Reshaping in On-Policy Optimization},
  author        = {Shaohang Wei and Zikun Su and Feifan Song and Wen Luo and Wei Li and Guangyue Peng and Houfeng Wang},
  year          = {2026},
  eprint        = {2608.00220},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  doi           = {10.48550/arXiv.2608.00220},
  url           = {https://arxiv.org/abs/2608.00220}
}

Download BibTeX · CITATION.cff

Code, scope, and reproducibility

The release supports inspection of training recipes, verifier implementations, evaluation protocols, non-visual analyses, and selected numeric tables. It is not a one-command reproduction of every reported result.

Included

  • verl/DAPO framework snapshot and reference recipes
  • Evaluation, verifier, aggregation, and unit-test code
  • Selected paper tables and curated static figures

Not included

  • Fine-tuned checkpoints and generated rollouts
  • Raw experiment logs and most intermediate CSVs
  • Site-specific launchers and general figure-rendering code
Repository Reproducibility matrix Evaluation configs Released tables Code license Third-party notices

Expanded paper figure