Inside the First Vision-Centric Parallel Thinking Framework for Vision-Language Models

Expanding inference width rather than length avoids exploration rigidity observed in vertically‑scaled paradigms. Prior models such as K2.5, Step3‑VL and LongCat‑Flash‑Thinking explored width‑oriented inference.

Visual‑Centric Path Division

Two visual‑centric partition strategies are defined:

Block division assigns each parallel reasoning path to a distinct image sub‑region (e.g., top‑left, top‑right, bottom‑left, bottom‑right), so the attention of that path concentrates on the designated quadrant.

Scan division gives each path a predefined visual scanning order (left‑to‑right, top‑to‑bottom, right‑to‑left, bottom‑to‑top), creating distinct attention trajectories across the whole image.

Block division can cause redundant computation in overlapping regions; scan division is structurally simple but may reduce path diversity. A hybrid training strategy mixes data generated by both divisions.

Core Mechanisms

Isolation : Path‑aware Attention introduces special <think_i> tokens that separate the context of each path, preventing attention leakage between paths.

Unbiasedness : All paths share the same position‑id interval during the parallel thinking stage. The summary token’s start position is set to the end position of the longest path plus one, eliminating positional bias.

Distinguishability : Learnable Parallel Rotary Position Embedding (LPRoPE) adds a learnable path‑specific absolute position embedding before applying rotary position encoding, preserving path identity while keeping the unbiased position range.

Training Data and Procedure

A parallel‑reasoning dataset of 163,000 question‑answer pairs is built from LVIS, LAION, Microsoft COCO, PixMoCount, RefCOCO, RefCOCO+ and RefCOCOg. Qwen3‑VL‑235B‑A22BInstruct serves as the teacher model. For each sample, four visual‑centric inference paths are generated using a hybrid of block and scan partitions at temperature 0.1. High‑temperature models Qwen3‑VL‑30B‑A3B‑Instruct and InternVL3‑5‑241B‑A28B provide additional diverse data and validation.

Experimental Results

Benchmarks focus on vision‑centric perception tasks:

Counting (PixMo, CountBench)

Visual search (V*)

Hallucination (MMVP, HallusionBench)

Grounding (RefCOCO)

On V* tasks the method improves scores by 12.6 points for the 3B model and 6.3 points for the 7B model. On HallusionBench the gains are 6.1 points (3B) and 5.0 points (7B). Grounding tasks show modest improvements over the baseline Qwen2.5‑VL.

Task‑Specific Preference Analysis

For counting tasks visual attention is distributed across the whole image; block division can cause overlapping calculations and hallucinations, so scan division is preferred. Block division offers explicit quadrant attention, while scan division changes the order of token attention, providing implicit diversification of reasoning paths.

Conclusion

Parallel reasoning substantially boosts performance on a range of visual perception benchmarks. Future work includes extending the framework with parallel reinforcement learning, multi‑round reasoning and agentic RL.

Paper: https://arxiv.org/abs/2602.13310

Code: https://github.com/xuhaoran1/Visual-Para-Thinker