AI agents are dominating tech industry conversations, from vendors to enterprise CIOs and boards of directors. But in IT, talk rarely translates to measurable value.
As vendors hawk the latest AI innovations, IT leaders are grappling with varying definitions of AI agents, technical immaturity and agent washing.
While enterprises are bullish on the technology’s potential, sifting through the hype and outsized claims is proving difficult.
“We had all the hype around generative AI, and then you had all these software companies that have to have something new to say, and so they say, ‘Well, now we have agents,’” Matt Kropp, managing director and senior partner at Boston Consulting Group, told CIO Dive. “There’s a fair amount of confusion out there.”
CIOs can start adding some clarity by asking vendors to define agents or be more specific about their agents’ capabilities.
“Agents are something that can perform some type of action using persistent memory and decision intelligence,” Tim Sanders, VP of research Insights at G2, told CIO Dive. “None of them are fully autonomous, so anybody that tells you differently is hiding something from you.”
Sanders said enterprises can consider agents as being on a gradient of weak to strong. Chatbots and copilots are on the weaker end, while a system of agents that works together would be on the stronger end.
“When you hear all of these stories about future workforces, what they’re talking about is the evolution of systems of agents,” Sanders said. “We’re not there yet, and I don’t think we’re close.”
Some of the confusion clouding enterprise judgement around AI agents also comes from the gap between vendor announcements and enterprise-ready availability.
“I’ve talked to 350 boards and executive teams over the last two years, and they’re all very excited about agents,” Kropp said, adding that clients want to know how the technology will change back-office departments. “All the software vendors that support all of the functions are building agents into their tools now, but they haven’t released much.”
Despite the thousands of vendors positioning themselves as agentic AI providers, few are delivering on their promises. Gartner estimates only around 130 among thousands of agentic vendors are “real.”
“We’re still early in the development of the tools, and because they’re early, most companies have not adopted them,” Kropp said. “Even once the technology is there, and the companies have adopted it, then you will have this issue of the people that are doing the work being resistant to adopting it.”
Gartner predicts that more than 40% of agentic AI projects will fail by the end of 2027 as businesses contend with escalating costs, unclear business value and risks galore.
Validating vendor AI claims
CIOs and enterprises that vet vendor claims will make informed purchasing decisions and reduce the risk of wasted resources
“I’m having a really hard time relying on vendor-supplied reliability statistics,” Sanders said. “Vendors can make things work really well in the lab, and then it gets into the real world and is messy.”
Validating AI agent claims is a bit different than other AI technologies.
Traditional AI models are evaluated on fixed datasets, while agentic AI assessments need to contain extended periods across various scenarios, “significantly” increasing the complexity, according to Eden Zoller, chief analyst of applied AI at Omdia.
“Agentic systems, particularly multi-agent frameworks, can exhibit emergent behaviors arising from dynamic interactions between individual agents and various external tools and data sources,” Zoller said in a blog post. “These system-level behaviors can emerge unexpectedly and present significant challenges for debugging and ensuring reliability.”
Analysts told CIO Dive that technology leaders should also question vendors about their product and capability roadmap around:
- Security and identity access controls
- User interface changes
- Agent interoperability
“We will probably have some failures where enterprises build agents or software companies build agents into their products that end up not being very usable,” Kropp said.
While vendor marketing tactics can result in short term disillusionment, the analysts CIO Dive spoke to agree that pulling back investments entirely could ultimately hurt companies down the road.
“We’re right to be skeptical today,” Kropp said. “But don’t mistake that for thinking agents are not going to have a massive impact.”
Disclosure: Informa, which owns a controlling stake in Informa TechTarget, the publisher behind CIO Dive, is also invested in Omdia. Informa has no influence over CIO Dive’s coverage.
