Open-Vocabulary Object Attribute Recognition with OvarNet: A Unified Framework for Detection and Attribute Classification

At CVPR 2023, the Xiaohongshu community technical team introduced a new task called Open‑vocabulary Object Attribute Recognition, which aims to locate, classify, and predict attributes of any object category in an image using a single model.

The proposed model, OvarNet, builds on large‑scale multimodal pre‑trained vision‑language models (e.g., CLIP) and employs prompt learning on available detection and attribute datasets. To achieve strong zero‑shot capabilities, fine‑grained class and attribute representations are extracted from massive image‑text pairs via weak supervision, and knowledge distillation is used to reduce computational complexity.

Key challenges identified include: (1) existing vision‑language models are biased toward object categories rather than attributes, causing feature misalignment; (2) a lack of datasets that jointly annotate bounding boxes, class labels, and attributes (COCO‑Attributes is limited); (3) the difficulty of training a unified framework that simultaneously performs detection and attribute classification in an open‑vocabulary setting.

To address these, the authors first built a two‑stage baseline called CLIP‑Attr, which generates region proposals with an offline RPN and matches visual embeddings with attribute word embeddings. Learnable prompt vectors are added to the text encoder, and the CLIP model is fine‑tuned on large image‑text corpora.

For efficiency, they introduced OvarNet, a one‑stage Faster‑RCNN‑style model that jointly performs detection and attribute prediction. OvarNet is trained on detection and attribute datasets and distilled from the CLIP‑Attr teacher to improve performance on novel/unseen attributes.

Experiments on VAW, MS‑COCO, LSA, and OVAD demonstrate that OvarNet achieves new state‑of‑the‑art results under both box‑given and box‑free evaluation protocols, confirming the complementary benefit of jointly learning object categories and attributes.

The source also contains recruitment information for algorithm engineering positions at Xiaohongshu, but the core academic contribution is the OvarNet framework for open‑vocabulary object‑attribute recognition.