Arize co-founder Aparna Dinakaran argues that agents have outgrown classical LLM-as-a-judge evals: as systems added tool calls, reasoning, and long-horizon sub-agents, they fail in ways fixed rubrics can't catch. Her answer is agent as a judge — adaptive, dynamic analysis — realized in Arize's new Signal agent, which reads traces, finds subtle failures like tool-call loops, and can even open a PR with a fix.
Aparna Dinakaran, one of the founders of Arize, opened by framing how far evals have come: from the new skill every PM and AI engineer had to learn, to the thing every serious AI team is now betting on. Arize gets a front-row seat into how the best AI teams build and ship agents, and into the eval teams running on live production agents via their traces.
The scale backs it up: Arize runs over 100 million evals every month, the average team runs about 12 different eval jobs, and the top teams run over 3,800 different evaluators. Offline and online evals each have their place, but Dinakaran focused on teams running evals on their traces — the data that helps teams see what's working, catch failures, and fuel continual learning loops. As she noted, the industry agrees, citing CPOs at Anthropic and OpenAI and Gary Tan echoing that evals are what you need.
The problem, Dinakaran argued, is that while teams built first-generation evals, the systems being evaluated shifted dramatically. In 2023 it was about answering a prompt. In 2024, frontier models added tool calls, reasoning, and deep research. Now teams run loops on real-world data with sub-agents kicked off on long-horizon tasks — each step a massive jump in complexity that produced a fundamentally different type of problem, not just a harder one.
Arize felt this directly through Alex, its own agent that lives in the product UI. As the frontier labs shipped new capabilities, Arize added them to Alex, giving it longer memory, the ability to create dynamic UIs, and search across an enormous volume of traces. But Alex would also forget context, fail to know when something was done, and sometimes get stuck in loops — failures the classical LLM-as-a-judge evals many engineers have written simply weren't enough to catch.
That pain led to the core revelation: what if the best way to evaluate an agent is actually with an agent? Dinakaran was careful to say this doesn't retire deterministic evals or LLM-as-a-judge — it adds a different tool for a different problem. Agent as a judge is about adaptive, dynamic analysis, whereas LLM as a judge gives a fixed rubric with fixed scores.
The distinction matters because Alex creates a new UI and a fundamentally different trajectory every time a user interacts with it. When trajectories differ on every run, a fixed rubric can't keep up. Her take: most teams today are doing the first two approaches, but the future of evals is having all three.
Dinakaran announced that Arize has released agent as a judge, along with Signal — a long-running agent that reads traces sent in and discovers patterns of issues that a classical LLM-as-a-judge eval with deterministic rubrics could never surface.
Signal has helped Arize find very subtle failures teams wouldn't think to look for, such as something running in a loop multiple times, the same tool being called repeatedly, or an inefficient trajectory. Because it holds all that analysis, it can go a step further and put up a PR with a fix. She closed by inviting attendees to the Arize booth (next to the OpenAI booth) for a demo, to the eval track in room 2005, and to a viewing party for that night's USA World Cup game.
We run over a 100 million evals every month.
What if the best way to evaluate an agent was actually with an agent
Agent as a judge is about adaptive dynamic analysis. LM as a judge just gives you a fixed rubric with these fixed scores.
most teams today are doing the first two, but the future of eval is actually having all three
| Time | Topic |
|---|---|
| 00:00 | Intro: Arize, the Evals track, and the future of evals |
| 00:31 | Evals as the bet every serious AI team makes; live production traces |
| 01:03 | Scale: 100M evals/month, 12 jobs per team, 3,800+ evaluators |
| 01:33 | Traces fuel continual learning loops; the industry agrees |
| 02:06 | What we evaluate changed: prompts to tool calls to sub-agent loops |
| 02:38 | Feeling the pain with Alex: memory, dynamic UIs, and getting stuck |
| 03:39 | The revelation: evaluate an agent with an agent |
| 04:44 | Releasing agent as a judge and Signal; booth, eval track, closing |