AI hallucination – why language models make up facts, how common it is and what your company can do about it

Definition

An AI hallucination is when a language model makes a claim that sounds credible but is invented or wrong. The model would rather guess than say "I don't know" – a business risk when AI gets your company, your prices or your service wrong.

What is an AI hallucination?

The word is borrowed from psychology and is actually misleading: the model does not experience anything. It works out an answer word by word, and when it has no material to go on, the made-up answer is just as well phrased as the correct one. For a decision-maker, the key point is that a hallucination does not look like an error. The text is fluent, the tone confident and the details specific. The error might be an invented quote, a product that does not exist, a wrong price or a founder who never existed. The term applies to generative AI in the broad sense: ChatGPT, Gemini, Claude, Perplexity and Google's AI Overviews can all hallucinate.

That makes hallucinations hard for the reader to spot – and that is exactly why they are a business risk, not a technical curiosity.

Why do language models hallucinate?

Language models hallucinate because they are built to predict the next word, not to check facts – and because training and evaluation reward guessing over saying "I don't know". An LLM has no internal database of verified facts. What it has is statistical patterns drawn from enormous amounts of text. When the question is about something the model has seen thousands of examples of, it usually gets the answer right. When the material is thin, the model fills the gap with whatever sounds most likely.

In September 2025, OpenAI published a research paper explaining the mechanism. Models are evaluated with tests in which a guess can score points, while "I don't know" always scores zero. That teaches the model to always answer – even when it has nothing to base the answer on. Source: OpenAI · Source: arXiv

The conclusion is simple: the less the model knows about a topic, the greater the risk that it makes things up. For most companies, their own brand is exactly that kind of topic.

How common is it for AI to make up facts?

Hallucination rates range from a few per cent to double figures, depending on the model and the task. Vectara's open leaderboard measures how often models add made-up details when they summarise a document. When measured in spring 2026, the best models scored in the low single digits, while others were clearly higher. Source: Vectara The figures change quickly as new model versions arrive – treat them as a snapshot.

The test also favours the models: they have the document in front of them. For an open question with no source material – "what does company X's service cost?" – the risk is higher.

In the courts, the effect is documented. The lawyer Damien Charlotin runs a database of court cases in which made-up, AI-generated case citations have been found. As of June 2026 it held more than 1,500 cases from around the world, with fines rising from a few thousand dollars in 2023 to six-figure sums in 2026. Source: Damien Charlotin

How do RAG and source citation reduce the risk?

RAG (retrieval-augmented generation) cuts the risk considerably by having the model take its answer from verified documents rather than from memory – but it does not eliminate hallucinations. The company's documents are stored in a vector database, the relevant text is retrieved for each question and the model is instructed to answer from it. Add citations – the answer pointing to its source – and users can check the claim for themselves.

The same mechanism drives public AI search. Perplexity, ChatGPT with web search and Google AI Overviews pull content from the web in real time and cite their sources. So what is on your website is the material the model answers from. If there is no clear answer there, the model falls back on guessing. How this works in practice is explained in What is AI search? and AI Overviews and GEO.

RAG therefore reduces the risk on two fronts: internally, in your own chatbots and assistants, and externally, by giving public AI services the right material to retrieve.

What happens if AI describes your company incorrectly?

When AI gets a company wrong, the company loses business without ever finding out. Hallucinations usually arise where training data is thin, and for almost every company outside the global giants, its own brand is one of those thin areas. The model fills the gap with what is typical for the industry, or with details about competitors: the wrong price level, a service you do not offer, an integration you do not have or a made-up founder. Source: Insightland

The most costly pattern is a different one. When the model lacks information, it tends to answer "no" rather than "I don't know". If a potential customer asks "does X integrate with Y?" and the answer is not clearly stated on your website, there is a real risk that the AI says it does not. The potential customer moves on to the next supplier. Nobody at your company sees the question.

The Air Canada case shows that the company is liable, too. The airline's chatbot invented a bereavement discount policy that did not exist. The customer bought a ticket believing the discount applied, and in February 2024 British Columbia's Civil Resolution Tribunal ordered the airline to pay CAD 812.02. Its argument that the chatbot was a separate legal entity was dismissed. Source: McCarthy Tétrault

Nor do the model providers offer a "correct the facts" button. In a GDPR case brought by the organisation noyb, OpenAI stated that incorrect information cannot be corrected in the model, only filtered out. Source: noyb All a company can influence is the material the model draws on.

How do you make sure AI describes your company correctly?

The most effective thing a company can do is make the right information easy to find, retrieve and cite – what is known in AEO as owned AEO. You do not control the model, but you do control what it finds when it looks. Four measures that usually make the biggest difference:

• A fact page about the company. A page that states plainly what you do, who for, where, how long you have been around, what pricing model you use and which integrations are available. Short, self-contained paragraphs: one question, one answer.
• Consistent entity information. The same company name, address, founders and description on your website, LinkedIn, industry directories and press material. Conflicting details are raw material for hallucinations; a clear, consistent picture of you as an entity gives the model something to anchor its answer to.
• Schema markup. Structured data for Organization, Product, Service and FAQ makes machines read facts as facts – not as text to be interpreted.
• Citable answers to buyers' questions. Take the questions your salespeople hear every week – "does it work with…", "what does … cost", "is it suitable for…" – and answer them clearly and completely. These are the paragraphs the AI services pick up.

Then keep measuring: once a month, put the questions you get from potential customers to ChatGPT, Perplexity and Gemini, and record the answers. Brand mentions in reliable external sources strengthen the picture further – the more sources agree, the less room the model has to guess. How this fits into a wider plan is described in AI strategy for companies.