‘The gap between AI ambition and infrastructure reality is widening’ Google Cloud report finds 83% of organizations must overhaul their infrastructure in order to maximize the agentic AI opportunity

• Most businesses are struggling to deploy agentic AI effectively, and legacy infrastructure is one of the key reasons, report finds
• Google polled IT leaders, with 83% stating that infrastructure upgrades are needed
• IT leaders are also concerned about the hidden costs of agentic AI, such as increased power consumption and operational complexity

If there was a single message to take away from this article, it’s that the infrastructure every business relies on today was not built to handle agentic AI.

Google surveyed over 1,400 senior IT leaders on their AI ambitions, and found that 83% of organizations say they require infrastructure upgrades to leverage the full benefits of production-grade agentic AI.

Moreover, many of those polled are also seeing unexpected costs arise from attempting to run agentic AI on legacy infrastructure. 62% said they had seen significant inference tax driven by data egress fees, storage bloat, and idle specialized hardware, alongside 82% who said that scaling AI introduces hidden operational complexity costs. 79% also reference security, governance, and MLOps as a key barrier to scaling agentic AI.

Upgrades needed for full agentic AI benefit

In order to combat these limitations, Google has several recommendations for organizations hamstringed by legacy infrastructure.

Leveraging fluid compute ‘to dynamically match the right silicon to the right task while minimizing operational overheads’ is Google’s first recommendation, providing compute power for agentic AI tasks without reducing capacity for general workloads, avoiding the need for excess memory usage to run agentic workloads that use large context windows.

For those battling agent sprawl caused by a cascade of new tasks across platforms and teams, Google recommends making use of enterprise-grade governance tools, which are usually available via the cloud partners businesses are already using. Google provides its own platform, Agent Gateway, as an example of a solution that provides visibility and oversight into how agents are communicating, the data they are accessing, and their workloads.

Organizing data more effectively prevents AI agents from drawing more compute when running heavy queries in attempts to access siloed data. Organizations looking to improve the efficiency of agentic AI should work towards using a unified data layer that automatically annotates unstructured data, allowing agents to understand where the data is without having to navigate pipelines. An added benefit of using a unified data layer is that it helps to avoid the duplication of data, saving on additional costs of storage bloat in the long run.

Moving your AI to the edge—by deploying agents directly on the site they are most used—is a further recommendation, and one that organizations are actively pursuing. 90% of organizations polled by Google said that this was a consideration in their AI initiatives. By deploying agents on site in manufacturing plants, retail stores, or hospitals, agents benefit from reduced latency, greater resilience (in the event of a centralized cloud outage), and improved cost-effectiveness by cutting per-token costs with local, highly optimized models.

As with businesses of all sizes, energy costs are a key consideration. When selecting new hardware, 91% of leaders now consider power consumption as a factor, especially when navigating power availability in regions without expanding capacity, regulatory compliance, and reducing the cost of ownership for AI systems.