John Ousterhout makes the case that AI workloads are shifting from throughput-bound bulk transfers to small, latency-sensitive message exchanges, and that TCP and RDMA are poorly suited to this world because they mismanage tail latency. He introduces Homa, a clean-slate Stanford transport protocol that is message-based, controls congestion from the receiver, and prioritizes short messages — reducing tail latency by an order of magnitude or more.
Ousterhout opens by arguing that AI's dependence on great networking is well established, but the shape of the workload is shifting. Historically AI traffic was dominated by enormous transfers — gigabytes of weight gradients and similar — where the only metric that mattered was throughput. Those workloads are easy for networks: transfers run so long that connection setup cost is irrelevant, and TCP and RDMA perform well.
Inference and agentic workloads, however, are becoming more granular, with smaller chunks of computation and small data exchanges for metadata and coordination — checking whether an entry is present in a distributed KV cache, or doing barrier synchronization at the end of a compute period. For these, roundtrip latency is what counts.
In a typical pattern, work is split across nodes that compute intensively on their GPUs, then do a small synchronizing exchange before the next round. While that exchange happens the GPUs sit idle, and because every exchange must finish before the next phase begins, a single slow exchange stalls the whole process.
When compute phases were about five seconds and exchanges took a few milliseconds, this didn't matter. But agentic workloads pushing tokens at a steady rate drive compute periods down to the millisecond scale, so millisecond synchronization now wastes a significant fraction of GPU resources. An audience poll drew more raised hands than Ousterhout expected — many people already see small-message latency limiting their throughput.
The root cause of tail latency is congestion from incast: several nodes send to the same destination at once, packets pile up at the top-of-rack switch's egress port, and a short message gets stuck behind long ones — in the worst case the switch runs out of buffer space, drops packets, and triggers timeouts and retransmissions.
In all protocols before Homa, including TCP and RDMA, congestion control is the sender's responsibility. Senders learn about far-away congestion late — historically through dropped packets, today via ECN marking relayed back through the receiver. Getting the rate right is very hard: multiple senders adjust simultaneously with only one bit of information and significant control lag, so the systems constantly oscillate and never stabilize. TCP's byte-stream model compounds this by hiding message boundaries, preventing prioritization of short messages and causing head-of-line blocking.
Homa is a clean-slate redesign for datacenter transport, begun as a PhD dissertation by Ousterhout's student Behnam Montazeri and turned into his personal project — a Linux kernel module, available on GitHub, that he is working to upstream. Its fundamental unit is a remote procedure call: a request message and a response message, so message lengths are known all the way down the transport.
Three properties follow. First, message-based rather than stream-based: as soon as the receiver gets the first packet it knows how much more is coming, and independent short messages can bypass long ones. Second, congestion control from the receiver, which has near-complete information and uses grant packets to pace scheduled packets and implement SRPT, favoring short messages. Third, Homa exploits the priority queues in modern switches (typically eight per egress port), dynamically placing short messages in higher-priority queues so they bypass queued long-message packets.
On a benchmark spanning message sizes from about 50 bytes to a megabyte, plotting roundtrip time against message length for both median (P50) and 99th-percentile (P99) latency, two results stand out. Short-message P99 is dramatically better for Homa — under 100 microseconds versus more than a millisecond for TCP, roughly 13x faster. And counter to intuition, long messages don't suffer: Homa is almost twice as fast as TCP even on the largest messages, thanks to run-to-completion scheduling rather than TCP's fair scheduling.
Ousterhout closes by predicting the role of small messages in AI will keep increasing, and urges the audience to ask whether high latency for short messages is capping their throughput. If so, Homa can likely cut tail latency by an order of magnitude or more. Now semi-retired from Stanford to spend all his time on the project, he offers to work directly with teams who want to try it.
latency matters and it's probably going to be mattering more in the future
congestion control is the responsibility of the sender
in Homa, virtually every major design decision is different from TCP and RDMA
Homa is basically my life mission right now
| Time | Topic |
|---|---|
| 00:47 | Latency matters more in AI — three goals for the talk |
| 01:50 | TCP and RDMA are poorly suited; introducing Homa |
| 03:24 | Workloads changing: small messages in inference and agentic work |
| 04:33 | Tail latency stalls GPUs; audience poll |
| 07:34 | Sender-side congestion control and why it fails |
| 11:11 | Homa: clean-slate, message-based RPC transport |
| 13:17 | Receiver-side congestion control, SRPT, priority queues |
| 15:54 | Benchmark results and closing call to try Homa |