WeChat Identify: From Object Detection to Large‑Scale Image Search – Technical Overview

WeChat Identify is an AI‑driven image recognition service that started as a simple object detector and has expanded into a full‑featured image search platform integrated into WeChat’s camera and scanning interfaces.

The system processes a query image by first performing background‑removing object detection, then retrieving a list of visually similar items, and finally extracting structured information such as title, brand, and main image.

WeChat’s classification framework uses a multi‑label, multi‑task backbone that simultaneously predicts image source (advertisement, photo, screenshot) and a hierarchy of 9 top‑level and 42 second‑level content tags, enabling fine‑grained understanding of complex scenes.

On the mobile side, a lightweight detection SDK employs optical flow to select stable frames, applies a gradient‑based sharpness filter, and runs a customized CenterNet model with a large receptive field and deformable convolutions, achieving comparable mAP to state‑of‑the‑art detectors with only ~1 M parameters and 25 ms latency per frame.

Server‑side detection leverages a dual‑stream RetinaNet built on Mask‑RCNN, providing both class‑wise object boxes for product categories and generic scene detection for natural images, facilitating rapid model updates.

Retrieval is powered by a large‑scale pipeline that indexes billions of images across multiple sharded databases; queries are routed to a small subset of relevant shards based on predicted categories, and a unified re‑ranking model merges results using an MLP that combines classification, detection, and similarity features.

For generic image‑to‑image search, an unsupervised clustering approach based on MoCo embeddings creates pseudo‑labels, forming 16 semantic shards; online routing selects the top‑3 shards for a query, followed by a multi‑shard re‑ranking stage.

The article concludes that the end‑to‑end pipeline—from data collection and semi‑automatic cleaning to model training, deployment, and continuous feedback—has enabled WeChat to deliver high‑quality visual search experiences across diverse scenarios such as product, plant, vehicle, and artwork recognition.