This research comes from the March 18 edition of the CB Insights newsletter. You can see past newsletters and sign up for future ones here.
As AI agents dominate the conversation, customers are growing skeptical about whether they can live up to the hype.
In March, we’ve interviewed 40+ customers of AI agent products and are hearing of 3 primary pain points right now:
- Reliability
- Integration headaches
- Lack of differentiation
1. Reliability
This is the #1 concern raised by organizations adopting AI agents, with nearly half of respondents citing reliability & security as a key issue in a survey we conducted in December.
According to CBI’s latest buyer interviews, AI agent reliability varies dramatically across providers. Many customers report a gap between marketing and reality.
“Whatever was promised didn’t work as great as said,” one LangChain user told us about the company’s APIs. “We encountered cases where we were getting partially processed information, and the data we were trying to scrape was not exactly clean or was hallucinating.”
For many customers, reliability is largely a function of how complex the data and use cases are. For instance, the LangChain customer saw ~80% accuracy for simpler tasks, but “for complex tasks, the accuracy dropped to around 50%.”
Organizations are tackling the reliability issue with 1) human oversight; and 2) more extensive model training.
An Ema customer, for instance, first has a subject-matter expert review outputs, and once “more than 90% of the responses that we have tested are now accurate, we let it fly.”
A customer for CrewAI, which orchestrates teams of AI agents into “crews,” takes an even more involved approach:
The customer still needs to intervene with their own ML algorithms when CrewAI is unable to handle outliers or unconventional data structures. If CrewAI is able to tackle these cases in the future, “that would be a huge leap forward.”
Source: CB Insights — AI agent market map featuring CrewAI and LangChain in the infrastructure category
2. Integration headaches
Integration limitations rank as another top customer pain point.
For one, lack of interoperability poses long-term challenges, as this Cognigy customer notes:
An Artisan AI customer echoes this: “It was a bit of a gamble that we were signing up for a product where they didn’t have quite all the integrations that we wanted.”
Where customers see real value from these tools is when they can support seamless data flow, especially through their existing tech stack. This buyer went with Decagon because of its integrations:
3. Lack of differentiation
More than half of private capital flowing into the AI agent space has gone to horizontal applications — but these markets, like customer support and coding, are becoming highly saturated.
“There’s so many short-term moats, but in the long term there is no moat,” one customer observed. “Whatever you build will be rapidly reproduced.”
In a crowded market, specialization will determine success.
Hebbia, for instance, has tailored its solution to financial players. An exec at a PE firm framed this as a selling point when getting internal buy-in: “When I bring tools to the deal team that live and breathe diligence and deal execution, ensuring that it’s aligned to what they know and understand and [that it] speaks their language is incredibly important.”
While many horizontal AI agents are actively deploying or even scaling their solutions, vertical AI agents remain nascent, with half still in the first 2 levels of Commercial Maturity.
They’ll gain more momentum this year as enterprises prioritize solutions that are highly tailored to the needs of individual industries.
CB Insights customers can read our latest interviews with AI agents’ customers here.
If you aren’t already a client, sign up for a free trial to learn more about our platform.