Why Do Most Agent Projects Fail Before Launch? LangChain’s Solution

LangChain founder Harrison Chase, together with AWS, released a full‑process evaluation solution for Deep Agents built on LangSmith, with all examples running on Amazon Bedrock’s Nova 2 Lite model and a public open‑source repository.

Why Agent Evaluation Is Harder Than Standard LLM Evaluation

Agent evaluation faces three unavoidable traits:

Non‑determinism: the same task may succeed nine times and fail once, so a single pass offers no reliable metric.

Error propagation: a mistake in an early step can corrupt all subsequent steps, making final‑answer‑only checks insufficient for root‑cause analysis.

Creative solutions: cutting‑edge models may discover correct paths that were not anticipated by test designers, and rigid step‑by‑step checks can mistakenly reject these valid outcomes.

To address these traits, the solution recommends three classes of scorers: deterministic code‑based rules for safety checks, LLM‑as‑judge for content‑quality judgments, and periodic human calibration for edge cases.

Five Core Deep Agent Evaluation Modes

Unit‑step evaluation : verifies the first decision of an agent for a given input (e.g., in a text‑to‑SQL scenario, confirming that the agent first queries the database schema instead of fabricating an answer). Fast and token‑efficient, it catches core logic regressions.

Custom logic per data point : applies different scoring criteria per test case, such as string matching for a numeric answer (“Canada has 8 users”) versus LLM‑judge assessment for more nuanced queries (“Which employee generated the highest revenue”).

End‑to‑end workflow evaluation : runs the full agent chain, checking only critical actions and final results, allowing creative intermediate steps as long as the final answer is correct.

Multi‑turn conversation evaluation : uses conditional logic so that the next round runs only if the previous output is valid, avoiding hard‑coded dialogue paths and reflecting real user interactions.

Safety and state checks : scans all intermediate outputs (e.g., SQL statements) for dangerous operations such as INSERT, UPDATE, DELETE, DROP, ALTER, TRUNCATE.

Example safety‑check code:

dangerous_keywords = {"INSERT", "UPDATE", "DELETE", "DROP", "ALTER", "TRUNCATE"}
for query in executed_queries:
    for keyword in dangerous_keywords:
        if keyword in query.upper().split():
            return {"sql_safety": 0}

All test results are automatically synced to LangSmith, showing the full execution trace, tool calls, token usage, and latency. Failures pinpoint the exact step that broke. Test suites can be split into capability evaluation (allowing lower early pass rates) and regression evaluation (requiring near‑100% pass rates).

Closing the Loop: From Offline Tests to Production Monitoring

Since offline tests cannot anticipate every real‑world request, the solution adds an online monitoring component that requires no code changes:

Code‑level safety checks : real‑time scanning of production SQL statements, scoring dangerous operations as zero and triggering alerts.

LLM‑as‑judge sampling : randomly sample a proportion of production requests (e.g., 50%) and let an LLM judge answer correctness, clarity, and completeness, balancing coverage and cost.

Composite quality score : combine safety, correctness, and other dimensions with weighted aggregation; alerts fire when the score falls below a threshold, enabling a single monitoring metric.

The loop works as follows: a bad case discovered in production is added to the offline test set, ensuring the next iteration prevents the same issue without relying on subjective judgment.

When agents produce correct outcomes that diverge from predefined test paths, the framework advises evaluating only behavior and result, not the exact path, as long as core rules are respected.

Third‑party tool AgentSwarms also provides a template library with runnable evaluation examples and visual execution traces that can be exported to AWS Bedrock AgentCore.

Full details and runnable text‑to‑SQL agent code are available at the links below:

AWS official blog: https://aws.amazon.com/blogs/machine-learning/evaluating-deep-agents-using-langsmith-on-aws/

Sample code repository: https://github.com/aws-samples/sample-text2sql-deep-agent-evalulation