Weekly Paper Notes — one of the top picks from the 2026-08-08 CS paper digest. Area: Distributed Computing / Computer Architecture.
Authors: Jirong Yang, Peizhe Liu, Jovan Stojkovic (UT Austin); Chaojie Zhang (Microsoft Azure) arXiv: 2608.04458 · PDF
TL;DR
Datacenter servers have been optimized for two workload shapes: CPU-centric services (web serving, key-value stores, analytics) and monolithic LLM inference, where a GPU does dense tensor math and the host merely feeds it. Agentic AI is neither. A single agentic request expands into a data-dependent graph of model calls, tool invocations, and orchestration decisions — and because orchestration and tools run on the host, the CPU lands squarely back on the critical path.
This paper is the first architectural characterization of that workload measured at production scale. The authors combine a fleet study across Microsoft Azure with a controlled study of open-source agentic frameworks (Owl, CORAL, Trae), organized by a taxonomy along three platform-relevant axes: how agents are orchestrated, how workflows are structured, and how models are composed. The findings are consistent and uncomfortable. Tool execution time has a heavy tail and exceeds inference time for more than 27% of fleet requests. A single CORAL run expands into 580 LLM calls interleaved with 552 tool invocations. Host CPU sits at an 11% median and spikes to near-100% at stage boundaries. Per-GPU power in a multi-role workflow shows four devices near peak while four idle at ~20%.
Their prototype, Agora, attacks all three mismatches: harvesting idle CPU cores for co-located throughput work (recovering 95% of the co-located workload’s standalone throughput, +30% host CPU utilization, <3% agent slowdown), oversubscribing GPU memory by consolidating agents (frees a third of the GPUs, +82% generation throughput, 2.5× lower tail latency), and pooling cores by software role with affinity-aware scheduling (up to 46% less tool CPU demand, 13% lower worst-case tool latency, 99% of serving throughput retained).
What problem is the paper actually attacking?
There is a large and healthy body of work on making LLM inference fast: continuous batching, PagedAttention-style KV-cache management, optimized attention kernels, disaggregated prefill and decode. Every one of those techniques optimizes the inference serving plane — the part of the system that turns a prompt into tokens.
There is also a fast-growing body of systems work on agentic frameworks: speculative agent execution that runs likely-next actions ahead of time, programmable software-defined serving that lets applications express and co-optimize the workflow, agent-aware schedulers that order workflow steps efficiently.
What sits between them — the machine — has gone unexamined. Nobody had asked what agentic workloads do to a server’s CPU, memory system, network, and microarchitecture, or whether the servers we buy are the right shape.
The motivating observation is the one visible in that diagram. In monolithic inference, the request crosses the CPU–GPU boundary twice. In an agentic workflow, it crosses hundreds of times, and each crossing is a synchronization point where one processor waits on the other.
The mechanism: a taxonomy, then a fleet measurement
The taxonomy is the load-bearing contribution, because “agentic workload” covers an enormous space and averaging over it produces nothing. The three axes:
- Orchestration mechanism — how control passes among agents (centralized coordinator vs. peer message-passing vs. shared state).
- Execution structure — sequential, parallel, or staged. This determines the shape of host load over time.
- Model composition — homogeneous (roles share one serving instance or use identical models) vs. heterogeneous (different model types or modalities). This determines how evenly the accelerator pool is used.
Each axis maps to a distinct resource pathology, and the paper demonstrates each one empirically.
Fragmentation → stranded capacity. A representative production request spans nearly a minute, alternating between multiple LLM calls, agent hand-offs, three tool-discovery operations, and three tool executions across two tools. Across the fleet, the distribution of time spent in tools versus inference is wide enough that “no single CPU-to-GPU time ratio characterizes the workload.” That’s a provisioning problem: whatever ratio you build the server at, most requests will strand one side.
Execution structure → bursty load. Trae is sequential at the agent level, but some reasoning stages fan out into parallel builds and test runs. Host CPU stays near an 11% median through the sequential portions and rises to nearly 100% at stage boundaries.
Decomposing host CPU into scheduler and tool components shows the structural signature clearly across frameworks: Owl activates only a subset of roles at a time, so scheduler demand stays low but fluctuates; CORAL runs four agents in parallel and holds scheduler demand steadily high; Trae alternates quiet intervals with near-saturating tool bursts.
Model composition → device imbalance. With each role served by a separate instance and roles differing widely in activity, role-level skew becomes GPU-level skew. In an eight-GPU Owl run, four GPUs serving the busiest text and vision roles repeatedly approach peak power while the other four sit near 20%.
Task diversity amplifies everything. Holding the CORAL framework and workflow configuration completely fixed and varying only the task set, research problems saturate host CPU at 99% while leaving GPU at 44%; algorithmic problems invert it, using 31% of CPU and 55% of GPU. Same framework, same taxonomy class, opposite bottleneck.
Why this doesn’t break existing infrastructure: the Agora prototype
The three architectural mismatches — stranded capacity from fragmentation, inefficient homogeneous CPU provisioning across three distinct host roles (scheduler, orchestrator, runner), and degraded microarchitectural locality from multiplexing agents onto shared cores — all admit software fixes on commodity hardware. That’s what makes the paper immediately actionable rather than a hardware wish-list.
CPU harvesting. Reclaim the temporal slack between bursts for co-located throughput work, but protect agentic tail latency when tools spike. The measured result: 95% of the co-located workload’s standalone throughput recovered, host CPU utilization up 30%, agent slowdown held under 3%.
GPU harvesting. Rather than dedicating exclusive GPU access to fixed agents, oversubscribe GPU memory by placing more agents per device, exploiting agents that share model state or are never simultaneously active, and prefetch the next agent’s state to hide swap latency. Result: a third of the GPUs freed, generation throughput up 82%, tail latency cut 2.5×.
Role-aware pooling. Partition host cores by software role instead of managing one homogeneous pool; isolate and right-size the control plane, and pin tasks within the runner pool to preserve cache and branch-predictor locality. Result: tool CPU demand down up to 46%, worst-case tool latency down 13%, 99% of serving throughput retained.
All three mechanisms are auto-tuned to the running workload — which follows directly from the characterization, since no static configuration suits workloads whose bottleneck flips with the task set.
Results
The numbers above are the results, and they’re worth reading as a set rather than individually. Each intervention is modest in isolation; together they say something structural: conventional uniformly provisioned servers leave roughly a third of accelerator capacity and a comparable slice of host capacity on the floor for this workload class, and the recovery is available without new silicon.
The microarchitectural finding deserves separate attention because it’s the least obvious. Multiplexing many agents onto shared cores degrades locality — agents evict one another’s cache lines and branch-predictor state, increasing pipeline stalls. This is the classic co-location interference story, but it appears here within a single application, between the concurrent agents of one workflow, which is not where most operators would think to look for it.
Why this matters
Agentic AI is projected to be among the fastest-growing consumers of datacenter capacity in the next few years, and it is being deployed onto machines designed for two other workloads. This paper is the measurement that establishes the mismatch is real, quantifies it against a hyperscaler fleet rather than a benchmark, and shows the recoverable fraction is large.
For anyone operating agent infrastructure today, the practical reading is direct: your GPUs are probably less busy than your dashboards suggest, your CPU is probably on the critical path in ways your monolithic-inference mental model doesn’t account for, and the three host roles in your stack want different machines. The role-based core pooling result in particular is something you could act on this quarter with cgroups and CPU affinity.
For architects, the forward-looking section is the more interesting one. The authors argue for dedicated hardware support to offload frequent scheduling and context-management operations off host cores, and for heterogeneous core types matched to host roles — efficiency cores for lightweight coordination, performance cores for compute-intensive tool execution. That’s a concrete, well-motivated ask for the next server generation, and it’s the kind of ask that only becomes credible after a paper like this one exists.
The obvious follow-up: this characterization is single-server. Agentic workflows increasingly span machines, and the network implications of hundreds of CPU–GPU boundary crossings per request, distributed across a cluster, are entirely unexplored here.
Read alongside
- Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (SOSP 2023) — the inference-plane optimization this paper explicitly positions itself beyond.
- Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) — continuous batching, the other half of the serving-plane baseline.
- Zhong et al., DistServe — prefill/decode disaggregation, the closest prior art in “split the workload across differently-provisioned resources.”
- Delimitrou & Kozyrakis, Quasar / Paragon — the classic datacenter co-location-interference lineage that the microarchitectural locality finding rejoins.
- Patel et al., Splitwise — power and heterogeneity in LLM serving fleets, and the closest methodological cousin for the production-fleet measurement approach.
Links
📄 arXiv abstract · 📄 PDF
Part of the Weekly CS Paper Digest series. Summary written from a close read of the preprint; figures cropped from the arXiv PDF and reproduced here under fair use for educational commentary.