Issues with Recommender System Benchmarks and Insights from the BARS Paper

This blog post discusses several problems with existing benchmarks in the recommender‑system field and introduces the viewpoints presented in the recent paper "BARS: Towards Open Benchmarking for Recommender Systems".

A benchmark is a community‑accepted standard used for comparison; in AI domains such as NLP and CV, a typical benchmark comprises a dataset, evaluation metrics, and a computation protocol, and it must be widely adopted by researchers.

The purpose of a benchmark is to provide trustworthy comparisons; without a common standard, results on obscure data or using unconventional metrics are difficult to validate.

Current recommender‑system benchmarks suffer from three major issues:

No universally accepted datasets and evaluation metrics. Recommendation is highly industrial, often split into recall and ranking stages, with billions of candidate items in recall. Public datasets rarely match this scale, creating a gap between academia and industry. Moreover, evaluation metrics are diverse, and the community confuses AUC (ranking) with Recall and NDCG (recall).

Irreproducibility. Many reported results cannot be reliably reproduced.

Lack of an easy‑to‑use framework. While RecBole provides a codebase for comparing common baselines, it does not constitute a full benchmarking suite.

The paper "BARS: Towards Open Benchmarking for Recommender Systems"—a joint effort by Huawei, Renmin University, Tsinghua University, and the Chinese University of Hong Kong—introduces an open‑source benchmarking tool ( https://openbenchmark.github.io/BARS/ ) and is recommended for beginners due to its clear language and comprehensive overview of recommendation research.

Key conclusions from the BARS study include:

Graph Neural Networks (GNNs) provide measurable improvements.

Every model category has at least one method ranking within the top‑5, indicating that recall tasks are not dominated by any single approach and there is ample room for innovation.

Methods leveraging item‑item similarity can yield large performance gains.

Simple models such as YouTubeDNN achieve strong results, while many more complex methods offer only modest improvements.

Additional findings on ranking (AUC) comparisons reveal:

No single model dominates all datasets, contradicting some prior claims.

Models such as DeepFM and xDeepFM perform competitively.

Achieving significant improvements in the ranking stage remains challenging.

These observations underscore the urgent need for a widely accepted benchmark; while BARS may not yet achieve universal adoption, it represents a crucial step toward more reliable and comparable recommender‑system research.