Arena co-founder and CTO Wei-Lin Chiang argues that as token flow shifts from chatbots to agents, the hard problem becomes measuring agent outcomes in the real world rather than on static benchmarks. Arena's new agent mode has collected over a million organic agentic traces, which it mines for explicit, implicit, and environment signals and combines — via a randomized-control-trial design — into a live leaderboard of model performance, cost, and token efficiency.
Wei-Lin Chiang introduced himself as co-founder and CTO of Arena, having done his PhD in AI research at UC Berkeley focused on building robust, scalable evaluations — work that became the foundation for Arena and included early efforts like LLM-as-a-judge (2023) and Chatbot Arena. He framed Arena as an AI evaluation company whose mission is to measure intelligence in the real world, beyond static benchmarks, capturing the intelligence that actually delivers value to users and customers.
Chiang walked through the AI breakthroughs Arena has tracked — from the ChatGPT moment and GPT-4 Turbo, to reasoning models like OpenAI o1, to image generation (including Nano Banana, which was tested in Arena under a code name before public release), and recent releases such as GPT Image 2, GLM 5.2, and the Fable breakthrough measured in agent arena. He cited Arena's scale: 10 million monthly visitors, 700 million collected conversations across text, vision, image, video, and coding/agentic modalities, and $100 million in annualized revenue just eight months after launching the evaluation product.
Chiang pointed to a rapid paradigm shift from chatbots to agents. Citing OpenAI's Codex data, he noted that the share of output tokens coming from agents has skyrocketed — essentially 100% of Codex output tokens inside OpenAI, with other organizations averaging above 60% — and that Codex adoption spans not just engineering but finance, recruiting, and legal at near-90% levels. With AI spending approaching human labor spend, choosing and optimizing the right model has never been more important.
Measuring agent outcomes, he argued, is the real bottleneck and a hard technical problem. Agents are multi-component systems — the model, the agent take loop, the tools, and the harness — where any piece can break the system. They operate through complex, long-horizon workflows (building apps, debugging, research, documents, slide decks), and the signals available in a trajectory become sparse and spread out, with a single task sometimes taking a hundred tool calls before success or failure is clear.
To deeply understand the problem, Arena decided to build a real-world agentic product itself to source organic traces and feedback from actual users. It launched an agent mode where anyone can go to arena.ai, choose the agent mode, and give a model a task — Chiang demoed downloading a Google Q1 earnings report and generating a PowerPoint slide deck, with the agent searching the web, writing Python, and producing a downloadable artifact. After each turn, users are asked whether the task was successful, providing a direct evaluation signal.
Under the hood, models are given a Claude-Code-like harness: file-system tools (read, write, edit), web search and fetch, image and speech generation, and terminal access to run code, with connectors like GitHub coming soon. In one week the tools logged 5.7 million tool calls, with bash the most-used at around 46%. In the first month, agent mode collected over a million agentic traces and generated more than 50 million lines of code across Python, Markdown, HTML, and JavaScript, with more than half of traces falling into professional, work-related categories — users building apps, debugging autonomous-vehicle control systems, and architecting RAG pipelines over hundreds of turns.
Chiang explained how Arena turns a million traces into a leaderboard by mining three types of signals: explicit signals where users directly report success or failure; implicit signals inferred from behavior such as downloading a file, praising, or complaining about output; and environment feedback about what actually happened when code ran or a command succeeded or failed. These are aggregated into signals like success rate, praise, over-compliance, durability, bash recovery, and hallucination — each producing its own ranking that is combined into the final leaderboard.
In the example shown, five different signals revealed that models perform differently across the board: Fable Five ranked number one with a net improvement of about 14% over the average, followed by Opus and GPT-5.1 High. Methodologically, the core idea is a randomized control trial that intervenes on an agent component to measure its causal effect on the outcome — general enough to also measure interaction effects between tools, harnesses, or system prompts. Because the leaderboard is built on real data, it can be sliced by task distribution (GDP-style professional work versus consumer use), cost (Fable at about $10 per session, plotted on a Pareto frontier), and token efficiency, where models with similar list prices can differ sharply in real-world token usage.
Chiang closed with practical guidance: if you are building an agentic app, log all agentic traces and interactions between the agent, the user, and customers; mine that data for insights; measure outcomes linked to the business metrics you care about; and use real-world data to choose the best model for your use case. He believes the best evaluation should be grounded and measured in real-world use cases rather than static benchmarks.
Looking ahead, Arena plans to add more connectors to bring in user context and enable evals for many kinds of agents (including coding agents on real repositories), incorporate more complex professional tasks sliced into categories, and develop richer signals and rubrics — even collaborating with users to define what good outcomes look like.
Our mission is to measure intelligence in the real world beyond just static benchmark
the token flow is now driven by agents
the core idea is basically a randomized control trial where we intervene on agent component
we believe the best evaluation should be grounded and measured in real world use cases like this
| Time | Topic |
|---|---|
| 00:00 | Intro: Wei-Lin Chiang and Arena's mission |
| 01:05 | Tracking AI breakthroughs, from ChatGPT to Fable |
| 03:12 | Scale: 10M visitors, 700M conversations, $100M ARR |
| 04:45 | Why agent evals matter: the shift to agents |
| 07:22 | Why measuring agents is a hard problem |
| 09:00 | Agent mode launch and live demo |
| 13:42 | From traces to a leaderboard: three signal types |
| 15:50 | RCT methodology, cost, and token efficiency |
| 18:59 | Takeaways for building agentic apps and what's next |