How to Rigorously Evaluate Large Models: Methods and Key Benchmark Datasets

Why Evaluation Is Needed

Direction for training – Loss reduction only shows memorization of training data; evaluation provides an objective measure of generalization.

Model selection and iteration – Quantitative scores enable comparison of different model variants and justify resource‑intensive training decisions.

Identify capability gaps – Systematic tests reveal weaknesses (e.g., poor math reasoning or code generation) that are invisible from informal probing.

Compliance and safety – Objective metrics are required to ensure that deployed models do not produce harmful, biased, or illegal content.

Evaluation Methods

2.1 Human Evaluation

Human annotators design test sets and scoring rubrics, then rate model outputs on dimensions such as quality, logic, fluency, and factual correctness. This captures creativity, humor, and user experience but is slow, costly, and can exhibit >30% inter‑annotator variance, limiting its use for large‑scale routine testing.

2.2 Benchmark‑Dataset Evaluation

Standardized test sets act as a “national exam” for models: the same questions and scoring rules enable reproducible, comparable results across models. Advantages are standardization, reproducibility, and fair horizontal comparison. Risks include data contamination (training exposure to test data) and possible mismatch between benchmark performance and real‑world behavior.

2.3 Automated Evaluation with a Judge Model

Stronger LLMs (e.g., GPT, Claude) serve as judges, scoring or directly comparing answers from the target model. This balances automation efficiency with the flexibility of human judgment but inherits the judge model’s biases, requiring careful prompt design and validation.

In practice the three methods are often combined: benchmark datasets for large‑scale testing, judge models for open‑ended tasks, and human checks for final quality assurance.

Typical Benchmark Datasets

3.1 General Knowledge & Language Understanding

MMLU – 57 subjects, 15,908 multiple‑choice questions; English knowledge benchmark.

MMLU‑Pro – more challenging version of MMLU.

C‑Eval – Chinese counterpart of MMLU, 13,948 questions covering 52 subjects.

CMMLU – 67 topics spanning basic to advanced Chinese knowledge.

3.2 Mathematics & Logical Reasoning

GSM8K – 8,500 elementary‑school math problems requiring multi‑step reasoning.

MATH – 12,500 high‑school competition problems, harder than GSM8K.

BBH – 204 difficult tasks across linguistics, common‑sense, software development, etc.

3.3 Code Generation & Software Engineering

HumanEval – 164 programming problems; metric is Pass@k.

SWE‑Bench – 2,294 real GitHub Issue‑Solution pairs; evaluates code patch generation and unit‑test passing. Variants include SWE‑Bench Verified and SWE‑Bench Multilingual.

3.4 Comprehensive Exams & Domain Knowledge

AGIEval – 20 official standardized exams (e.g., college entrance, legal, math competitions) assessing cognitive ability.

GPQA Diamond – 448 expert‑crafted multi‑choice questions in biology, physics, chemistry.

For a general‑purpose model, covering at least MMLU (English), C‑Eval (Chinese), GSM8K (math) and HumanEval (code) provides a balanced capability profile. Domain‑specific benchmarks should be added for specialized applications.

Evaluation Tools

4.1 Dataset Testing with EvalScope

EvalScope is an open‑source lightweight framework from ModelScope that supports over 20 built‑in benchmarks (MMLU, C‑Eval, GSM8K, MATH, HumanEval, SWE‑Bench, etc.) and custom CSV/JSONL datasets.

Built‑in datasets – plug‑and‑play without manual download.

Custom datasets – upload your own question‑answer files.

Automatic metric computation – accuracy, BLEU, ROUGE, Pass@k, etc.

Visualization & comparison – Web UI with arena mode.

Multi‑model compatibility – local HuggingFace models, OpenAI‑compatible APIs, ModelScope models.

Two testing modes are provided:

Public benchmark testing – specify model and dataset name, e.g., mmlu or gsm8k. EvalScope loads the data, runs inference, and computes scores automatically.

Custom QA testing – provide a formatted dataset, optionally weight multiple datasets to obtain a composite score for domain‑specific models.

Custom datasets are typically kept between 100–500 high‑quality samples; quality matters more than quantity.

4.2 Prompt Design for LLM‑as‑Judge

When using a judge model, prompt engineering determines evaluation accuracy. The workflow consists of:

Define evaluation dimensions (e.g., relevance, correctness, completeness, friendliness, safety).

Design a scoring rubric (e.g., 1–5 scale with clear descriptions for each score).

Write judge prompts that embed the dimensions, rubric, and required output format; include few‑shot examples if needed.

Implement an evaluation script that loads the target model and the judge model, generates answers, invokes the judge with the prompts, aggregates scores, and produces a report.

Validate and iterate by manually inspecting a subset of judge scores, adjusting prompts or rubric, and repeating until results are stable.