Comprehensive Survey of Agent Harness Engineering Unveils a Seven‑Layer Framework

Introduction

The paper Agent Harness Engineering: A Survey (CMU, Yale, JHU, Virginia Tech, Amazon, etc.) provides the most systematic and engineering‑focused overview of how to run agents beyond the model layer. It catalogs more than 170 open‑source Agent Harness projects and demonstrates the evolution from Prompt Engineering to Context Engineering and finally to Harness Engineering.

Why the System Matters More Than the Model

The authors observe that academic research has long focused on model capabilities, but when agents tackle long‑running tasks, failures are often caused by the surrounding system rather than the model’s intelligence. For example, a study that only changed the tool‑format and surrounding harness (leaving the model untouched) achieved up to a 10× boost on an encoding benchmark. A fixed GPT‑5.2‑Codex agent, after redesigning the system prompt, adding middleware context injection and self‑validation hooks, rose from 52.8 % to 66.5 % on Terminal‑Bench 2.0 . Meta‑Harness, using automatic harness optimisation, reached 76.4 % on the same benchmark, surpassing hand‑crafted solutions.

同一个模型，换一套执行外壳，表现可以完全不一样。

These results suggest that a strong model is often already sufficient; the bottleneck lies in tooling, context management, sandboxing, verification, and permission systems.

Three Evolutionary Stages

Stage 1 – Prompt Engineering focuses on crafting system prompts, few‑shot examples, and step‑by‑step reasoning. The engineering target is a narrow input text.

Stage 2 – Context Engineering addresses what the model should see at each step: selective information, memory retrieval, tool result compression, and handling of full‑window overflow for long‑running tasks.

Stage 3 – Harness Engineering moves the bottleneck outside the model: state maintenance, tool orchestration, permission enforcement, feedback injection, trace logging, and failure recovery.

In short, Prompt Engineering asks “how to talk to the model”, Context Engineering asks “what the model should see”, and Harness Engineering asks “how to make the model act reliably in the real world”.

ETCLOVG: A Seven‑Layer Harness Framework

Execution : Where does the agent run? Local, container, browser, desktop, remote sandbox? What are the boundaries?

Tooling : How are tools described, discovered, invoked, and protected from misuse?

Context : Management of short‑term context, session state, and long‑term memory.

Lifecycle : Single‑round vs. multi‑round loops, planner‑executor‑reviewer division of labour.

Observability : Tracing model calls, tool calls, retrievals, errors, retries, token cost, latency.

Verification : Checking correctness, diagnosing whether failures stem from model, tool, context, or test environment.

Governance : Permission checks, audit trails, identity management, and safety gates.

Only when all seven layers are combined can an agent reliably execute long‑duration tasks.

Observability and Governance as First‑Class Concerns

The survey argues that observability and governance should be treated as independent layers rather than after‑thought add‑ons. In production, agents not only call tools, run shell commands, modify code, or access databases—they act in the world. Without full observability you cannot diagnose failures; without governance you cannot safely deploy agents even when they succeed.

Rethinking Evaluation: Trace‑Native Metrics

Traditional benchmarks report only final success rates (e.g., “52.8 % pass”). The authors show that such numbers hide many variables: model, prompt, tool set, context, sandbox, retry policy, permission, and evaluator. They propose a trace‑native evaluation that records the complete execution trace—including model outputs, tool calls, environment state changes, errors, retries, token usage, latency, and cost—and then judges three aspects: result correctness, path reasonableness, and evaluator trustworthiness.

This shifts evaluation from a simple leaderboard to a quality‑control mechanism that can pinpoint why an agent failed and which harness layer needs improvement.

Competitive Landscape Shifts to the Harness

The survey identifies three cross‑layer tensions:

Cost‑quality‑speed triangle: stronger sandboxing, finer permissions, and richer traces improve safety but increase cost and latency.

Capability‑control paradox: more tools and longer memory expand functionality but raise prompt‑injection risk and privacy concerns.

Harness coupling: changes in one layer affect others (e.g., tool descriptions consume context window, execution environment influences evaluation results, traces lacking identity data cannot support governance).

Consequently, optimisation is never local; improving prompts, tools, memory, sandbox, or verifier can alter the whole system’s behaviour.

From Frameworks to Platforms

Historically, the race was to build the quickest agent loop (framework level). The survey predicts a move toward full Agent Platforms that provide durable workspaces, managed sandboxes, identity, billing, observability, evaluation, governance, and human hand‑off. A timeline figure shows the evolution from 2022 to 2026, illustrating the transition from isolated frameworks to integrated platforms.

In practice, developers should stop asking only “which model is stronger?” and start evaluating the surrounding harness: execution environment stability, tool interface design, context drift, state recovery across rounds, trace‑based debugging, verifier coverage, and closed‑loop permission/audit mechanisms.

Conclusion

Prompt Engineering awakens the model, Context Engineering shows it the right information, and Harness Engineering makes it act reliably in the real world. The next competitive frontier for agents is not model capability but the quality of the engineering shell that surrounds it.