Critical Examination of Face Recognition Benchmarks and Overstated Accuracy Claims

2014 marked a breakthrough year for face recognition, with reported accuracies on the LFW dataset climbing from 97.25% (DeepFace) to 99.15% (DeepID2), yet these figures often ignore the underlying methodological flaws.

First, many papers compare algorithms using different training data and out‑of‑set images, resulting in unfair comparisons that violate the principle of fixed datasets for fair evaluation.

Second, reported performance differences are rarely subjected to statistical significance tests such as ANOVA or t‑tests, leaving it unclear whether observed gains are due to random variation or genuine improvement.

Third, claims that algorithms surpass human performance are questionable because no proper metric exists to compare a single algorithm against the diverse decision models that constitute “human” recognition.

Fourth, the focus on raw accuracy overlooks the actual requirements of target applications; for entertainment uses modest accuracy suffices, while security‑critical biometric systems demand verification rates at extremely low false‑accept rates, which current methods fail to achieve.

Fifth, the hype around “fully automatic” deep neural networks ignores that their architectures and training procedures are heavily handcrafted and rely on numerous heuristic choices, and that traditional models like logistic regression and SVM still embody valuable theoretical insights.

The author concludes that amidst the excitement surrounding deep learning, a sober, statistically sound, and application‑aware approach is essential for genuine progress in AI.