Qwen3-VL-Seg Unlocks Pixel‑Level Open‑World Segmentation

1. Bounding boxes are just the start

Multimodal large language models (MLLMs) can generate bounding boxes, but these are too coarse for dense visual tasks such as robotic grasping or fine image editing. The authors ask whether an MLLM can directly output pixel‑level masks without relying on heavy external segmenters.

Qwen3-VL-Seg addresses this by treating the predicted box as a structural prior and designing a lightweight, box‑guided mask decoder that follows a four‑step coarse‑to‑fine decoding strategy:

Multi‑scale spatial feature injection to enrich intermediate visual features needed for dense prediction.

Spatial‑semantic query construction that fuses box geometry with language features into an object query.

Boundary‑guided pixel fusion that injects fine‑grained texture while suppressing background clutter.

Iterative mask‑aware query refinement where the first‑pass mask is fed back to progressively sharpen boundaries.

2. Complex instructions amplify the advantage

To evaluate open‑world capabilities, the team tests on in‑domain (RefCOCO‑style) and out‑of‑domain samples, covering single‑instance, multi‑instance, phrase, and descriptive commands, as well as extreme scales, severe occlusion, harsh lighting, and category shift.

Results show that when instructions become more complex—e.g., "segment the third red car on the left partially hidden by a tree"—Qwen3-VL‑Seg’s superiority over baselines becomes markedly larger.

Across tasks such as referring expression segmentation, visual grounding, and open‑world segmentation, Qwen3‑VL‑Seg reaches state‑of‑the‑art performance on RefCOCO benchmarks, excels on language‑dense commands, and generalizes well to out‑of‑distribution scenarios.

3. Open‑world segmentation dataset

Beyond model architecture, the authors build a massive training set called SA1B‑ORS, derived from SAM’s SA‑1B data. It contains two complementary subsets: a class‑level referring set (e.g., all cats) and a description‑level instance set (e.g., the collie wearing a collar on the left), covering a spectrum from simple to complex references.

4. Dialogue‑style image segmentation

The paper also discusses dialogue‑style segmentation, where prompts may require functional, relational, or physical reasoning to answer intent‑driven questions such as "which suitcases can be removed without disturbing the stack?" This task benefits from the fine‑grained mask capability of Qwen3‑VL‑Seg, offering a more concise solution than previous SAM + QwenVL pipelines.

Open‑source address: https://arxiv.org/pdf/2605.07141