What Agent Harness Do AI Phones Like OpenAI’s AI Phone and Gemini on Android Really Need?

PhoneHarness: Mixed‑Action Orchestration Harness

PhoneHarness is a research platform that enables mobile agents to choose among three action surfaces—device‑side CLI commands, GUI interactions, and host‑side MCP‑style tools—based on the sub‑goal of a task. The core hypothesis is that a useful phone agent must not only tap screens but also select the appropriate surface to produce verifiable side effects such as file creation, setting changes, or calendar entries.

Mixed‑Action vs. Traditional GUI‑Centric Evaluation

Traditional evaluation treats a phone task as a screenshot → tap/swipe/type pipeline, which works for single‑app, low‑side‑effect scenarios. PhoneHarness models a task as a workflow that may span CLI, GUI, and MCP tools, shifting the evaluation focus from visual appearance to whether observable side effects truly occurred and whether the execution trace is auditable.

Example workflow: search for information inside an app, supplement it with a web search, compose an email, and verify the email was sent. This requires GUI interaction, external retrieval, text processing, and a verifier that checks the email object.

PhoneHarness Bench: Verifiable Benchmark

PhoneHarness Bench builds on the harness by defining each benchmark task with:

a user goal,

a set of callable action surfaces, and

a task‑specific verifier that checks side effects.

During execution the agent records screenshots, CLI/MCP operations, file changes, system state, and app‑side results. The verifier then decides whether the evidence chain supports successful completion, rather than relying on the model’s textual claim.

Demo Scenarios

CLI‑first: Query device status via CLI, then decide whether to launch a GUI flow.

Mixed workflow: Retrieve data with an MCP tool, perform GUI actions, and let a verifier re‑check the result.

Virtual display: Execute GUI steps on a background virtual display while preserving a trace.

Experimental Findings

Experiments show that performance gains stem from mixed‑action routing rather than improved GUI clicking. Significant improvements appear on tasks with deterministic paths, tool‑assisted steps, or verifiable side effects—e.g., device‑status queries, file handling, web retrieval, and calendar/email/document workflows. Pure GUI‑heavy tasks still suffer from visual grounding issues, permission dialogs, login state variability, and unstable search results.

The results suggest that future mobile agents should focus on selecting the right action surface and providing auditable execution traces instead of merely scaling GUI click models.

Implications for the AI‑Phone Era

OpenAI AI Phone and Gemini on Android illustrate a shift from app‑centric to agent‑centric devices: users express high‑level goals, and the agent schedules actions, invokes tools, manipulates apps, and produces verifiable outcomes. Realizing this paradigm requires infrastructure such as mixed‑action harnesses and verifiable benchmarks.

PhoneHarness expands the agent’s action space to match real‑world mobile workflows, while PhoneHarness Bench ensures that evaluation reflects genuine task completion.

Resources

Paper: https://phoneharness.github.io/assets/paper.pdf

Project homepage: https://phoneharness.github.io/

GitHub repository: https://github.com/PhoneHarness/PhoneHarness

HuggingFace dataset: https://huggingface.co/datasets/PhoneHarness/phoneharness-bench