A year ago the hard problem was getting an agent to find and click a button; now the hard problem is reliability — doing end-to-end work that lives "in the seams" of many apps where there is no unit test to check the result. The Amazon AGI Lab's answer is perception agents that read the rendered screen (not the code behind it), confirm their own output instead of firing and hoping, and let you point at what you want changed. The team open-sourced the first two pieces of its harness — an annotation Chrome extension and a verification tool that checks work against a design spec.
A year ago the hard problem was getting an agent to find a button and click it on a screen it had never seen; now agents can drive browsers and are starting to drive desktop apps. But the speaker argues clicking turned out to be the easy part. The real work — onboarding a new hire by setting up accounts, adding them to Slack, booking intros, ordering laptops — is an end-to-end process nobody owns that touches five different systems.
Agents can usually perform each individual step, yet they still struggle to complete the whole job because "the real work lives within the seams" between applications. The agent can use every tool you give it and still can't finish the work — which is where it all falls apart.
Tool use, chaining agents, and operating workflows are all capabilities, and the field has mostly figured out how to add capabilities. The next hard part is reliability — and without reliability there is no trust. As a gut check: an end-to-end agent workflow that succeeds 60–80% of the time sounds fine, but if it deletes a database one time in four you will never touch it again. Real trust needs reliability "in the nines."
The one place enormous progress on reliability and trust already happened is coding. Autocomplete became function-writing became agents that write code and open their own pull requests, to the point that hardly anyone reads every generated line anymore. Coding made that jump because code is verifiable: you can run it, test it, and be sure it worked. Reliability showed up first wherever the answer could actually be verified.
Most knowledge work is not like code. Did the report land? Is the design on brand? Did it capture what you actually meant? There is no unit test for those questions, so verification hits a wall exactly where most work lives — in the seams of the apps people use day to day. Nobody has cracked how to make an agent reliable when the answer can't be easily verified.
Humans handle this messy work by figuring things out together — grabbing a colleague, jumping on a call, pointing at systems, and solving it in minutes even though none of it is directly verifiable. A key ingredient is looking at the same screen: shared context means much less explaining. The claim is that agents don't need a bigger brain so much as this shared context.
Today's agents can see a screen and take actions, but once they fire off an action they move on — they don't watch what happens or recover when a step goes sideways. The speaker borrows the robotics loop: perceive, then plan, then act. Perception has to come first — taking in the rendered screen the way a person does, and keeping up in real time rather than the prompt-wait-retry rhythm inherited from chatbots. The goal is an agent that can react while you are still working.
Perception agents matter for three reasons. First, they complete the loop on computer use: a perception agent reads the rendered screen and can confirm its own output instead of acting and hoping. Second, they need no API or backend because they work off the rendered interface — the same pixels and structure you see — which matters because most software exposes no API. Third, input flows the other way too: instead of writing a long description, you can point at a heading or section, a far more precise and less lossy signal the agent can act on exactly.
The team just launched the first two pieces of its perception-agent harness as open source. The first is annotation — a Chrome extension that lets you select elements on a screen (draw around a heading, hover to find the right element, click to select) and issue changes like "make it red" or "double the font size." It captures the location, style, and feedback into a complete summary you hand to the agent, removing the back-and-forth because the agent sees exactly what you saw.
The second piece is verification: you describe your design rules in a design.md file (or let the tool infer them), and the agent checks its own work against those specs. It runs a visual check (on brand, correct layout) and a user-flow check — actually walking through the app, adding or deleting a task like a real user — then writes a report calling out what passed and what didn't, so you don't have to click through everything at midnight.
Perception isn't only visual. In a live experiment the speaker and a colleague, Giovanni, held a design meeting wearing Bee wearables (thanked as sponsors) that transcribed the conversation. The transcript and its captured insights were pulled in and applied to a website with one click — turning the background yellow, the heading red, and swapping an emoji — automatically kicking off verification, which flagged whether the changes stayed within the approved design rules. The speaker framed this as only the very first step, with the rest being built in the open, and asked the audience to try the GitHub repos and give feedback — "none of us get smart alone."
We taught computers to use computers.
if your agent one in four times deletes a database, you will never touch that agent again
So reliability showed up in the first place you can actually verify the answer.
none of us get smart alone and that's the whole point. We want to build AI that makes all of us smarter together.
| Time | Topic |
|---|---|
| 00:00 | Clicking was the easy part; the real work lives in the seams |
| 02:38 | Capabilities are solved — reliability is the next hard problem |
| 05:14 | Coding got reliable first because code is verifiable |
| 06:45 | Messy knowledge work: verification hits a wall; humans share context |
| 08:20 | Perceive, plan, act — the loop today's agents skip |
| 12:40 | Open-source harness demo: annotation and verification |
| 16:18 | Perception beyond screens: Bee-device meeting experiment and call to action |