Top AI Models Achieve Under 4% Task Completion in Real-World SaaS Benchmarks

SaaS‑Bench Overview

The SaaS‑Bench benchmark, released by UniPat AI and Peking University, evaluates leading large‑language‑model agents on 23 real‑world SaaS applications covering six professional domains: software engineering & project management, business operations & finance, medical administration, team collaboration & document workflow, agricultural supply chain, and independent media creation.

It contains 106 realistic tasks (74 pure‑text, 32 multimodal). 93 % of tasks require interaction with at least two applications , and more than half need three‑way hops. The average number of UI actions per task exceeds 100 steps, with some tasks approaching 400 steps.

Each run executes inside an isolated Docker container that freezes software versions, databases, and user credentials. Agents interact solely through the rendered DOM using mouse and keyboard; no back‑door API or database access is permitted.

Scoring Methodology

Two strict scoring schemes are applied:

Checkpoint score : tasks are decomposed into dozens of verifiable sub‑steps; each correctly completed sub‑step contributes partial weight.

Solve score : an all‑or‑nothing rule—any single sub‑step error yields zero points.

Claude Opus 4.7, the current state‑of‑the‑art model, achieves an average checkpoint score of ~44 % and a solve rate of only 3.8 %.

Failure Patterns Identified

Analysis of failure logs reveals four recurring defects:

Domino‑style cascade failures : a single mis‑named entity propagates downstream errors. Example bof_032 requires creating a corporate client “Arcturus Digital” with two invoices. The model also fills a personal‑name field, causing the system to treat the entity as a personal client; subsequent accounting steps fail because the expected corporate record is absent.

Blind overconfidence : the agent logs an intended correction (e.g., changing an invoice date) but never verifies the change, leading to a false‑positive score.

Extreme volatility : identical tasks produce widely varying outcomes because minor UI choices or extra clicks exhaust the operation quota.

Long‑chain probability decay : with an estimated 95 % success probability per step, a 12‑step chain yields less than 55 % chance of full success, explaining the steep drop‑off in solve scores.

Allowing up to three retry attempts modestly improves both checkpoint and solve scores, confirming that stochastic factors play a non‑trivial role.

Implications for Software Design

The benchmark highlights that current agent designs focus on pixel‑level UI manipulation while ignoring the underlying business‑logic loops that define real work. To enable reliable digital employees, enterprise software should expose clear, verifiable actions and state changes—removing unnecessary menus, hidden panels, and lazy‑load effects that act as obstacles for machines.

Agents must not only trigger UI events but also confirm that server‑side state has been updated (e.g., re‑querying a database record after clicking “Confirm”). This closed‑loop verification is essential for handling long‑chain workflows across multiple applications.

References

https://unipat.ai/blog/SaaS-Bench

https://github.com/UniPat-AI/SaaS-Bench

https://arxiv.org/pdf/2605.15777