Defining a Good Answer in the Agent Era: A Rubrics Survey

As large language models evolve from simple QA to deep research, medical consulting, multimodal generation, and long‑term agent tasks, evaluating output quality becomes increasingly difficult because many scenarios lack a single correct answer or a clear verification signal.

Rubrics address this gap by breaking the abstract concept of a "good report" into explicit items such as factual correctness, coverage, evidence support, reasoning clarity, usefulness, safety, format compliance, and others. Evaluators or judge models can score each item, providing fine‑grained feedback that can be turned into training signals.

The survey formalizes a rubric set as a collection of rubric items, each consisting of a natural‑language description and an importance weight. It distinguishes rubrics from related notions: LLM‑as‑Judge decides *who* evaluates, rubrics decide *what* criteria to use; reward models output a single scalar, whereas rubrics enumerate multiple explicit standards; RLVR relies on automatically verifiable answers, while rubrics suit open‑ended tasks that cannot be fully verified.

Four construction paradigms are identified:

Direct generation : a powerful LLM generates a full set of criteria given a task description, candidate answer, or reference evidence.

Contrast generation : the model compares a high‑quality and a low‑quality answer and extracts discriminative standards.

Iterative optimization : rubrics are refined through cycles of validation, decomposition, and filtering to produce more atomic and compact sets.

Online/co‑evolution : for reinforcement‑learning and agent tasks, rubrics evolve alongside policy roll‑outs, incorporating newly observed failure modes.

In model training, rubrics convert complex quality requirements into optimizable supervision. For policy‑model training, a judge model scores each rubric item, aggregates the scores into a reward, and feeds it to PPO/GRPO or other RL algorithms. Advanced rubric rewards may include learnable weights, veto or saturation mechanisms, curriculum based on difficulty, and integration of environment feedback.

Rubrics also improve reward‑model training by making the model’s preferences interpretable (showing why a response is better), providing finer‑grained supervision (separating factuality, safety, etc.), and enabling the construction of higher‑quality training data that focuses on substantive dimensions rather than superficial cues.

For evaluation, rubrics serve as explicit standards for open‑ended tasks. In general‑purpose benchmarks they are used for reasoning, deep research, open generation, agent capabilities, and alignment assessment. In domain‑specific settings (medical, legal, finance) rubrics check factual correctness, safety risk, professional expression, and practical usability of both intermediate trajectories and final answers.

The survey highlights several open challenges: (1) reward hacking, where models exploit superficial rubric features without genuine quality improvement; (2) limited generalization of rubric‑based reward models across tasks and domains; (3) bias introduced by rubric authoring or judge‑model selection; (4) personalization and safety concerns, as customized rubrics may conflict with safety standards or become attack vectors.

In conclusion, rubrics constitute an explicit, structured quality interface that links human preferences, task constraints, and model optimization, making it possible to move beyond vague intuition toward measurable, diagnosable, and improvable model behavior.