Warp, the AI-native terminal company, has published a playbook for building coding agents that learn from their own mistakes, according to Anthropic. The approach centers on tight feedback loops, evaluation harnesses, and Claude as the reasoning engine. This matters because most teams ship agents that plateau after launch. Warp ships agents that get sharper every week.
What stands out here is the method: it’s less about fancier prompts and more about building infrastructure that turns every agent run into training signal. Here’s the step-by-step guide to replicate it.
Quick Start
You’ll learn how to design an agent loop that improves itself through evaluations, failure analysis, and prompt iteration. You need: Claude API access, a task domain with measurable outcomes (coding, support, research), and the discipline to log everything.
Step 1: Define the Agent’s Job Narrowly
Start with a bounded task. Warp didn’t build a general assistant. They built agents for terminal workflows like debugging commands, writing scripts, and navigating codebases. Narrow scope means you can measure success clearly. If you can’t write down what “good” looks like in one sentence, you can’t evaluate it.
Step 2: Build the Evaluation Harness First
Before shipping a single agent response, build the scorer. Warp runs agents against a curated set of real tasks with known correct outcomes. Each run produces a pass/fail, plus diagnostic data on where it went sideways. Skipping this step is the number one reason agents stop improving. You need a ruler before you can measure growth.
Step 3: Pick Claude for the Reasoning Core
Warp uses Claude because agent work requires long-horizon reasoning, tool use, and the ability to recover from dead ends. Per Anthropic, Claude’s extended thinking and tool-use reliability are what make self-correction feasible at scale. Use the strongest model your budget allows for the reasoning steps. Cheaper models for classification and formatting.
Step 4: Instrument Every Run
Log the full trace: prompt, tool calls, intermediate reasoning, final output, and eval score. No aggregated metrics. Raw traces. When an agent fails, you need to watch the replay, not read a summary. This is the raw material for improvement.
Step 5: Cluster Failures, Don’t Chase Them
Don’t patch individual bugs. Group failures by root cause. Warp categorizes by failure mode: wrong tool selected, bad argument format, hallucinated file path, gave up too early. Fix the category, not the instance. One prompt tweak can resolve fifty failures if you diagnose right.
Step 6: Iterate Prompts Against the Eval
Every prompt change is a hypothesis. Run it against the full eval set. If the score drops, revert. If it climbs, ship it. No vibes-based prompt engineering. This is where most teams stall because they change prompts based on the last bug they saw, not the aggregate.
Step 7: Feed Production Data Back Into Evals
Real users find edge cases your synthetic evals miss. Every production failure becomes a new eval case. The harness grows. The agent has to clear a higher bar each release. This is the self-improving part: the system ratchets up automatically.
Step 8: Version Everything
Prompts, tools, model versions, eval sets. When a metric drops, you need to know what changed. Warp treats agent configs like code with full git history and rollback paths.
Why This Matters
The gap between agent demos and agent products is this exact loop. Demos work on handpicked inputs. Products face the full chaos of real usage. Warp’s method is the discipline layer that turns one into the other. Expect this to become the default playbook as more companies move from chatbots to agents that actually do work.
Next Steps
Start with ten eval cases in a spreadsheet. Wire up Claude to run them. Log traces to a file. That’s the minimum viable loop. Expand from there. Full technical details are available at the original Anthropic source.