AI Hallucination

A hallucination is the failure mode where the agent sounds right and is wrong. It can show up anywhere the model is allowed to fill a gap: a refund window that does not exist, a clinic hour that was never set, a tracking number assembled from plausible digits. In a chat or voice agent it is the primary quality risk, because unlike a dropped call it leaves no obvious trace until a customer acts on the wrong information.

How it works. A language model generates the most probable next word given its prompt. If the prompt contains the fact, the probable continuation is the fact. If it does not, the probable continuation is whatever text usually follows that kind of question in training data, which is a generic, confident answer. The model has no internal flag that separates recalled text from invented text; both come out at the same fluency. Hallucination risk therefore rises whenever the question is specific to your business and the supporting passage is missing, outdated or buried among too many others.

Why it matters for compliance and revenue. A hallucinated promise is a promise your business made. 'Yes, your plan covers water damage from a burst pipe' from an insurance agent is a claims dispute waiting to happen. 'The dentist can see you Saturday' when the practice is closed is a no-show and a one-star review. In healthcare and finance, a wrong statement about coverage or eligibility can be a reportable incident. The cost is asymmetric: hundreds of correct answers earn trust slowly, and one invented policy destroys it in a screenshot.

Example. A restoration company's after-hours chat agent is asked whether it bills the insurer directly. The policy document covers this, but the uploaded version is a scanned PDF the indexer could not read. The retriever returns nothing useful. Without a guardrail, the model writes, 'Yes, we bill your insurance company directly and you pay nothing out of pocket', which is false for the deductible. With a decline rule, it says it cannot confirm billing terms, offers the office number, and logs the question as unanswered. The next morning the owner replaces the scan with a text PDF and the question answers correctly.

Hallucination vs wrong retrieval. If the agent quotes an outdated price that really was in an old document, that is a retrieval or content hygiene problem; the model did its job faithfully. A hallucination is the model producing content no source supports. The fix for the first is the knowledge base; the fix for the second is prompt rules, decline paths and tools that read live data.

How to test for it.

• Ask questions whose answers are not in your documents and confirm the agent declines rather than answers.
• Ask about prices, dates and coverage with slightly wrong premises ('my plan includes weekend visits, right?') and check that it corrects you.
• Sample 50 transcripts a week and trace every factual claim to a passage or tool result.
• Track the unanswered-question log; a growing list is a content gap, a shrinking one means gaps are being closed properly.

Botcadence agents are grounded on your uploaded documents and hand off to a person with the full transcript when a question falls outside them; see chat agents and human handoff.