What is an AI hallucination?
An AI hallucination is output from a language model that is presented confidently but is false or unsupported: an invented figure, a misquoted standard, a citation that does not exist or a summary that adds something the document never said. It happens because the model generates likely-sounding text rather than looking facts up.
Also calledhallucinationconfabulationAI error
Compare the softwareBest AI accounting software →All 22 bookkeeping and accounting vendors →
- What it is
- Confident but false AI output
- Main cause
- Generating text without a source
- Main control
- Cited sources and human review
Why AI hallucinations happen and how to control them
Language models predict the next words from patterns in their training data. When a question goes beyond what the model can answer from the material in front of it, it can still produce a fluent answer, which is why hallucinations are hard to spot. In finance the risks are specific: a wrong number in variance commentary, a paragraph reference to a standard that says something else, or a contract term summarised incorrectly. Controls include grounding the model in retrieved documents, requiring citations for every statement, limiting the tool to defined tasks, checking figures against the ledger rather than the model's text, and having a person review before anything is posted, sent or filed. Quillon ties each statement in its technical accounting memos to the source paragraph so reviewers can check it.
Common questions
Why do AI models hallucinate?+
They generate the most likely words from patterns they learned, not from a lookup of facts. Without source documents to work from, a fluent but wrong answer can look just like a right one.
How can finance teams reduce hallucinations?+
Use tools that answer from retrieved documents and cite them, check numbers against the ledger, restrict AI to defined tasks and keep a reviewer before anything is posted or sent.
Are hallucinations an audit risk?+
Yes, if AI output enters records, disclosures or audit evidence unchecked. Firms and finance teams set review steps and keep the sources an AI-drafted document relied on.
Software that cites its sources
Part of Accountio’s accounting technology coverage · Glossary
