Large language model (LLM) tech has been in the making for years, but earnings call transcripts show unprecedented interest in LLMs from executives for the first time in 2023, fueled by the meteoric rise in popularity of ChatGPT.
Executives are discussing the use of LLMs in a wide range of applications, from risk analysis to content generation to medical imaging.
Interviews conducted by CB Insights with software buyers, as well as Analyst Briefing surveys submitted to CB Insights by startups, show enterprise budget allocations for LLMs entering the millions.
Here are 3 key takeaways from buyer transcripts:
- There’s no clear winner yet in foundation models: Despite OpenAI’s dominance, buyers are still closely evaluating other model developers against key criteria like latency, ethical response, and token limits before making purchasing decisions.
- Open-source models are competing with proprietary models on price: Some buyers have indicated that open-source models like Meta’s Llama2 have comparable performance and lower deployment costs, resulting in better ROI for the organization.
- Domain-specific models are emerging to compete with general-purpose models: Buyers are considering models trained on industry-specific datasets like clinical notes and financial data for domain-specific tasks as an alternative to models trained on general text.
Below, we’ll highlight several key LLM vendors, how much their buyers are paying, and vendor-specific buyer perspectives.
How much are buyers paying LLM developers?
Based on interviews with buyers and survey data submitted by startups, we found that companies’ annual spend values typically range from $15K to $5M.
Buyer perspectives
Buyers have highlighted a number of factors driving vendor selection as well as renewal appetite, including privacy and token limits.
OpenAI
Azure integration gave OpenAI a leg up with this healthcare buyer.
“…these [domain-specific] models are not getting trained… on the volume of data anywhere as close to what ChatGPT is trained on… It [OpenAI deployment] was pretty minimal overhead… the goal… is to essentially enable the use of ML tools that are available from the Azure subscription at the Enterprise level…” — VP, Machine Learning, Fortune 500 company
Read the full transcript here.
This buyer evaluated open-source alternatives.
“…Meta’s invention… it may be cheaper than the OpenAI [model] because it’s open source. Then we believe the performance of the Hugging Face and also the Llama-2 is also comparable to the OpenAI [model]. Maybe just a little bit weaker than that, but maybe the overall…ROI is quite a good deal.” — Cloud Data & AI Lead, Fortune 500 company
Read the full transcript here.
aI21 labs
Pricing will influence this gaming startup’s intent to renew.
“For the vendor to get to a 9.5 or 10 satisfaction score, they would need to continue to compete on cost level. OpenAI just lowered their costs by 10x this past week, so that’s a big change… [AI21 Labs] would need reinforcement learning as a must; that’s what ChatGPT is doing so well.” — C-level Executive, VC-backed entertainment company
Read the full transcript here.
anthropic
This buyer chose Anthropic for higher safety standards.
“Strengths, I would say, the ethical considerations of privacy and bias, fairness… their model outperformed the other models, including GPT-3 and ChatGPT… In terms of weaknesses, the specificity of the model output and the interestingness of the model output… I think that other weakness also was in terms of speed and efficiency, like latency, and once you ask a question, how long does it take to fully respond.” — Senior Manager of Data Science, $1B+ valuation technology company
Read the full transcript here.
If you aren’t already a client, sign up for a free trial to learn more about our platform.