1 views collected around this technical thread.
This article introduces the concept of modality, outlines the five research branches of multimodal models, and presents a concrete case where multimodal deep‑learning techniques are applied to predict and improve game frame rates using both static and temporal features.
The seefun team leveraged face detection scores and quality metrics to denoise and weight‑fuse facial features during training and testing, using a three‑layer MLP with Swish activation and dropout, and achieved a 0.8983 mAP (fourth place) on the iQIYI‑VID‑2019 video person‑recognition challenge.
The ‘One Name’ team from Nanjing University achieved a MAP of 0.8986 and third place in the iQIYI multimodal video retrieval challenge by fusing official face embeddings with scene features, using channel‑attention‑based video feature fusion, a multimodal SE‑ResNeXt module, and a carefully partitioned model ensemble.
The authors introduce a feature‑fusion pipeline and a Residual Dense Net that leverages multi‑frame face embeddings to identify persons in iQIYI‑VID‑2019 videos, achieving 0.9035 mAP (second place) with only ≈0.5 GFLOPs and processing the full test set in minutes.
Watchdog team won TOP1 in iQIYI & ACMMM2019 multimodal video person recognition challenge using pre‑extracted multimodal features, a 2048‑dim classifier with BCE loss, re‑ranking, DALI‑accelerated re‑detection, fine‑tuned InsightFace, and multi‑model ensembling achieving ~91% test accuracy.
iQIYI’s large‑scale hierarchical classification system combines multimodal text and image embeddings, low‑rank multimodal fusion, and a dense hierarchical multilabel network with cascade‑style weighting to assign accurate type tags to short videos, boosting production efficiency and personalized recommendation diversity.
The DFS face-detection algorithm jointly created by Didi AI Labs and Beijing University's PRIS team secured five first-place and one second-place results on the WIDER FACE benchmark, achieving 96.3% (Easy), 95.4% (Medium) and 90.7% (Hard) AP by leveraging a Feature Fusion Pyramid and semantic-segmentation supervision, and is already deployed in Didi's driver-identity verification and in-vehicle privacy systems.