Ptechhub
  • News
  • Industries
    • Enterprise IT
    • AI & ML
    • Cybersecurity
    • Finance
    • Telco
  • Brand Hub
    • Lifesight
  • Blogs
No Result
View All Result
  • News
  • Industries
    • Enterprise IT
    • AI & ML
    • Cybersecurity
    • Finance
    • Telco
  • Brand Hub
    • Lifesight
  • Blogs
No Result
View All Result
PtechHub
No Result
View All Result

The physics of agentic scale: Why full-stack co-design matters

By CIO Dive by By CIO Dive
November 16, 2026
Home Enterprise IT
Share on FacebookShare on Twitter


An AI agent spends much of its life waiting, which can carry a substantial infrastructure cost. A conversational chatbot typically waits for a person between turns. An agent can initiate the next step as soon as a tool returns, creating a sustained sequence of dependent requests. A chatbot handles one prompt at a time, and the pause before the next one is a person thinking, so the serving layer can treat each request as self-contained and scale by adding replicas.

An agent resolving a customer’s billing dispute works in a loop: It reads the ticket, calls the billing system, waits, reasons again, calls another tool, and then responds. Each pause is a machine waiting on another machine, each turn depends on the one before, and tasks can run for minutes or hours.

The infrastructure has to hold the agent’s state across those pauses and keep its context ready for the next step. Buying infrastructure for agents as if they were chatbots means organizations are paying to wait.

Figure 1. Each successive turn depends on the results of the previous one

Source: Omdia

 

Why faster chips alone do not solve the problem

Adding parallel hardware does not remove dependencies between turns. The waiting creates three distinct costs.

  • Latency. Ten four-second database queries add 40 seconds to a task. A faster accelerator does not make those queries return sooner.
  • Accelerator memory. While an agent waits, its conversation’s working memory (the KV cache) can remain on the accelerator, occupying capacity other requests could use. It can move to slower memory for later retrieval or be discarded and rebuilt, repeating computation.
  • Idle compute. An accelerator doesn’t have to wait for one agent to finish its task, as the serving software can handle other requests in the meantime. An agent’s execution environment can still reserve CPU and memory while it waits for a tool response.

Figure 2. Adding compute capacity does not eliminate the costs of waiting

Source: Omdia

 

According to recent Omdia research, 81% of organizations stated that agentic AI is a strategic priority across IT and cloud, and 80% reported that ensuring positive ROI for their AI initiatives is a top concern.1 Omdia research also found that 81% of organizations considered scaling AI agents into full production more complex than they initially expected, while 61% identified agent costs as a major barrier to adoption.2

Four questions every CIO must answer before agents reach production

Full-stack co-design refers to engineering chips, memory, networking, storage, serving software, and orchestration together around the workload. Four questions show why evaluating accelerators alone misses where the money goes. Google’s AI Hypercomputer is the example for each.

  1. Does the gateway know what the model already holds in memory? The best server for an agent’s next turn is often the one that already holds the conversation’s context because another server may need to retrieve or rebuild that context. A gateway that knows this routes accordingly, and one that does not makes the organization pay for unnecessary rebuilding across turns. For example, GKE Inference Gateway routes this way, and Google reports that its predictive routing cuts time to first token (TTFT) by up to 70%.
  2. What happens to an agent’s state while it waits? An agent that executes generated code needs its own isolated environment, and an environment that sits idle while a tool call returns still costs money. The decision is whether to pay for idle agents or to suspend them and accept a short delay when work resumes. For example, GKE Agent Sandbox provides the isolation, and Google reports allocating 300 sandboxes per second at sub-second latency. Agent Substrate, released as open source in May 2026, suspends idle agents and restores them with sub-second responsiveness. It is available on GKE for evaluation and non-production use.
  3. Which stage of the loop runs on which silicon? Most of an agent’s loop is orchestration and tool calls, which run well on ordinary CPUs. Inference accelerators handle the model’s prompt processing and token generation, and training hardware handles model improvement. Running the whole loop on the most expensive hardware means paying accelerator prices for work a CPU could do. Google’s eighth-generation TPUs come in a training variant (TPU 8t) and an inference variant (TPU 8i), both expected later in 2026, alongside Axion CPUs and NVIDIA GPUs. Framework support eases moving workloads onto TPUs, but portability still has to be tested with the buyer’s own models.
  4. Where does the agent’s context live, and how fast does it get back to the accelerator? Long-running agents can accumulate more context than available accelerator memory can hold. Retaining that context for reuse can require moving some of its KV cache to cheaper memory or storage. The slower that round trip, the longer the agent waits before its next inference step. In Google’s tests, keeping the KV cache in cheaper memory tiers cut TTFT by 44% for 10,000-token prompts. Offloading context to Managed Lustre shared storage cut GPU hours by nearly 60% for 50,000-token prompts.

These decisions interact: Keeping context close may speed the next turn but consume memory, while releasing resources can introduce restoration delays. Co-design makes those trade-offs explicit across the system instead of optimizing each component independently.

Figure 3. Co-design balances routing, agent execution, hardware selection, and context storage

Source: Omdia

 

How to run the proof

The performance figures above are Google’s own. CIOs should ask for reproduction on their own workloads, including the costs of retaining, moving, and rebuilding state. The four questions double as the test plan. Measure cache hit rates at the gateway and resume times for suspended agents, useful work per hardware pool, and time spent waiting on storage.

Co-design within one provider reduces integration work but can create another silo if components only work well together there. Open components, including llm-d and Agent Substrate, help address that concern. Whether they deliver comparable performance elsewhere remains the test.

The procurement question, therefore, changes: Instead of asking which accelerator wins a benchmark, organizations should measure successfully completed tasks per dollar at the required quality and response time. Vendors should answer with measurements from the buyer’s own environment.

Learn how full-stack system co-design powers the next generation of AI agents with AI Hypercomputer.



Source link

By CIO Dive

By CIO Dive

Next Post

ZTE Powers High Efficiency AIDC with Robust Energy Infrastructure

Recommended.

Goldman Sachs picks China stocks poised to benefit from a new wave of AI-related hardware exports

August 16, 2026
Hammerspace CEO On Its New AI Data Platform For Nvidia AI Environments

Hammerspace CEO On Its New AI Data Platform For Nvidia AI Environments

March 17, 2026

Trending.

Kioxia launches 245TB LC9, the biggest flash drive on the market | Computer Weekly

Kioxia launches 245TB LC9, the biggest flash drive on the market | Computer Weekly

July 28, 2025
OpenAI Really Wants Codex to Shut Up About Goblins

OpenAI Really Wants Codex to Shut Up About Goblins

April 29, 2026
Hyperscaler capex ballooned last year, fueling AI bubble fears

Hyperscaler capex ballooned last year, fueling AI bubble fears

January 5, 2026
The Coolest Big Data System and Platform Companies Of The 2026 Big Data 100

The Coolest Big Data System and Platform Companies Of The 2026 Big Data 100

June 9, 2026
China’s AI Boyfriend Business Is Taking On a Life of Its Own

China’s AI Boyfriend Business Is Taking On a Life of Its Own

January 20, 2026

PTechHub

A tech news platform delivering fresh perspectives, critical insights, and in-depth reporting — beyond the buzz. We cover innovation, policy, and digital culture with clarity, independence, and a sharp editorial edge.

Follow Us

Industries

  • AI & ML
  • Cybersecurity
  • Enterprise IT
  • Finance
  • Telco

Navigation

  • About
  • Advertise
  • Privacy & Policy
  • Contact

Subscribe to Our Newsletter

  • About
  • Advertise
  • Privacy & Policy
  • Contact

Copyright © 2025 | Powered By Porpholio

No Result
View All Result
  • News
  • Industries
    • Enterprise IT
    • AI & ML
    • Cybersecurity
    • Finance
    • Telco
  • Brand Hub
    • Lifesight
  • Blogs

Copyright © 2025 | Powered By Porpholio