Interpretation of the Paper “Multi-View Multi-Label Learning with View‑Specific Information Extraction” (SIMM)

IJCAI 2019 was held in Macau from August 10‑16, receiving 4,752 submissions and accepting 850 papers (17.9% acceptance). Alibaba Youku Lab had five papers accepted. This article interprets one of those papers titled “Multi-View Multi-Label Learning with View‑Specific Information Extraction”, authored by Southeast University and Alibaba’s Youku AI platform.

The paper addresses the problem of learning from objects that have multiple diverse representations (e.g., an image described by HSV histogram, Gist, SIFT, etc.) and multiple semantic labels. Traditional multi‑label methods either concatenate all feature vectors (causing high dimensionality and over‑fitting) or sum them element‑wise (incompatible dimensions). Multi‑view multi‑label learning (MVML) is proposed to integrate heterogeneous views effectively.

The authors argue that existing MVML methods focus only on shared information across views, ignoring view‑specific (private) contributions. They illustrate this with examples where certain labels are directly linked to specific views (e.g., the label “pink” is derived from HSV, while “flower” comes from Gist).

To overcome these limitations, the paper introduces SIMM (View‑Specific Information Extraction for Multi‑view Multi‑label learning). SIMM jointly extracts a shared subspace and view‑specific information using a neural‑network framework. The overall loss consists of a multi‑label loss L_ml and an adversarial loss L_adv that confuses a discriminator D about the origin view of the shared representation, encouraging the shared subspace to contain no view‑specific cues.

The shared subspace loss also includes a multi‑label component L_sml to preserve semantic meaning. The combined shared‑subspace loss is:

For view‑specific feature extraction, the authors define a private feature vector s^v (extracted by view‑specific encoder E^v ) and enforce orthogonality between s^v and the shared vector c via a specific loss L_specific :

The overall architecture (Figure 2) jointly optimizes all modules during training. At test time, given an unseen example x^* , the prediction is obtained by the final output layer (Figure 3).

The experimental section evaluates SIMM on eight multi‑view multi‑label datasets (six benchmarks plus a Youku video annotation set). Six baseline algorithms are compared, including two SIMM‑related baselines, ML‑kNN variants, and two MVML methods (F2L21F, LSAMML). Six standard multi‑label metrics are used: Hamming Loss, Average Precision, One Error, Coverage, Micro‑F1 (higher is better for AP and Micro‑F1, lower is better for the others). Results show that SIMM achieves the best performance in 87.5% of cases and second best in 10.4%.

Further analysis varies the balance parameters α and β (setting them to zero removes the shared and specific losses). Figure 3 demonstrates that without these constraints performance degrades on Pascal and Youku15w, confirming the benefit of jointly modeling shared and private information.

In summary, SIMM introduces a novel MVML framework that simultaneously optimizes an adversarial shared‑subspace loss and a view‑specific orthogonal loss, leading to consistent improvements over baselines across multiple datasets and metrics.

References include works on adversarial multi‑task learning, ML‑kNN, multi‑label learning surveys, and recent MVML methods.