Why is my RAG hallucinating?

Text extraction is lossy and fails differently per format. Scanned or image PDFs yield nothing without OCR. Multi-column layouts interleave into word salad. Tables collapse into space-separated runs where the row-and-column association that carried the meaning is destroyed. DOCX text boxes, PPTX speaker notes, secondary spreadsheet sheets and PDF form fields are commonly skipped by default extractors. The failure is silent — you get a shorter string, not an error.

What you'd see

• ·Prose questions answer well; table, figure and number questions fail.
• ·Failures cluster in one document class — scanned contracts, spec sheets, decks.
• ·The system behaves as though a document you can see in front of you doesn't exist.

Confirm it

1. Grep the stored chunks Fetch the stored chunk text for the source document and search for the fact verbatim. If it isn't there, stop — this is ingestion, not retrieval.
2. Sort by characters per page Compute extracted characters per page across the corpus and sort ascending. Scanned pages sit near zero and stand out immediately.
3. Render a parsed table back as text Check whether a human can still tell which value belongs to which column. If not, neither can the model.

If confirmed, fix

• medium Layout-aware parsing, with OCR triggered by a characters-per-page floor
• medium Convert tables to Markdown or HTML with headers preserved
• high Per-format pipelines instead of one universal extractor
• low Hard ingestion assertions — minimum chars per page, must-contain probes — that fail loudly

Often confused with

retrieval-miss — Identical user-visible symptom ("it can't find it"). One grep of the stored chunks separates them, and the fixes have nothing in common.

Two sub-modes. Splitting — the fact spans a boundary, so neither chunk answers alone and each retrieves at mediocre similarity. Orphaning — the chunk is intact but its referents live elsewhere: a table row without its header, "the above limit applies to…", a pronoun whose antecedent was in the previous chunk, a clause under a "Section 4: Terminated Employees" heading the chunk no longer carries. The model then attaches a correct number to the wrong entity.

What you'd see

• ·Retrieval returns the right region of the right document and the answer is still half right.
• ·Values get attributed to the wrong row, product, region or period.
• ·Fixed-size chunking with no overlap; long enumerated policy documents fail worst.

Confirm it

1. Read the top-k chunks as a human Could a competent person answer from these alone, with no other knowledge? This single check resolves most "the model is hallucinating" reports.
2. Look at where the gold fact sits Facts at the extreme start or end of a chunk are the ones being severed.
3. Check for header-less table fragments A table fragment without its header row is meaningless to the model and to you.

If confirmed, fix

• medium Split on structure (headings, sections) rather than character count
• low Prepend a breadcrumb — doc title > H1 > H2 — to every chunk's embedded text
• medium Keep tables atomic, or repeat headers on every fragment
• medium Small-to-big retrieval — embed the small chunk, feed the parent section

Often confused with

ingestion-loss — Here the text is present in the index and readable. There it never made it in at all.

Three separate leaks. Update leak — incremental sync keyed on a timestamp the source doesn't reliably bump, or a partial failure retried at batch level that skipped documents. Delete leak — the pipeline only ever upserts, so there are no tombstones and retracted policies stay retrievable forever. Reindex leak — the embedding model or chunking strategy changed and only new documents were written under the new regime, leaving the index internally inconsistent.

What you'd see

• ·Answers are correct-but-outdated rather than invented.
• ·Failures correlate with recently edited documents.
• ·It cites documents that were archived months ago.

Confirm it

1. Diff a recently edited document Pick one edited in the last 7 days, query the index by its stable id, and diff stored text and updated_at against the source.
2. Query for a deleted document If its content still comes back, you have no tombstone path.
3. Compare document counts Index count versus source count, plus the age distribution of indexed_at.

If confirmed, fix

• medium Idempotent upserts keyed on a stable document id plus a content hash
• medium Propagate deletions explicitly — tombstones, not silence
• medium Treat freshness as a monitored SLO (p99 source-to-index lag), not an assumption
• low Gate a full reindex on any embedding-model or chunking change

Often confused with

corpus-conflict — Stale index means the current version isn't retrievable. Corpus conflict means both versions are legitimately present and the system picked the wrong one.

Vector search with metadata pre-filtering intersects two conditions. If metadata was never populated at ingest — null tenant_id, missing doc_type, absent effective_date — a filter that looks perfectly reasonable silently matches nothing. ACL filters built from a stale permissions snapshot do the same. Strong pre-filters can also over-constrain the approximate search so it returns fewer than k results, or degrades toward random.

What you'd see

• ·Works for admins, fails for regular users. Works for one tenant, not another.
• ·"I can't find it" for a document the user has open in another tab.
• ·Recall varies by role or tenant rather than by question.

Confirm it

1. Run the query with all filters removed If the gold chunk appears, the filter is your answer.
2. Log the filter clause actually sent Log what reached the vector store, not what you think you built.
3. Measure null rates on filterable fields A field that's null for 30% of the index is a filter that drops 30% of it.

If confirmed, fix

• low Validate metadata at ingest and fail on nulls in filterable fields
• medium Post-filter with over-fetch where the permission surface is small
• low Log filters with every query
• medium Per-role and per-tenant recall tests in CI

Often confused with

retrieval-miss — The retriever is working perfectly here — the fix is in the data plane, not the search configuration.

Real corpora contain v1 and v2 of the same policy, a draft beside the signed final, US and EU variants, an internal wiki that contradicts the handbook, and an FAQ nobody has touched since 2021. Both documents are genuinely relevant, both embed near the query, and no reranker can adjudicate — because relevance is not the axis on which they differ. Authority and effective date are. This is a data-governance failure being misdiagnosed as a model failure, and it is the single most commonly misattributed cause on this list.

What you'd see

• ·The "hallucinated" text exists verbatim somewhere in your corpus.
• ·Two subject-matter experts disagree about whether the answer was even wrong.
• ·Wrong answers are internally coherent and cite a real, existing document.

Confirm it

1. Grep the corpus for the bad answer Search for a distinctive phrase from it. If it's there, you do not have a hallucination problem — you have a corpus problem.
2. Retrieve k=20 and count contradictions How many retrieved chunks assert mutually incompatible claims?
3. Check whether authority metadata exists at all Is there an effective_date, a version, a status field? Often there isn't.

If confirmed, fix

• medium Designate canonical sources and exclude everything else from the index
• medium Tombstone superseded versions
• medium Add effective-date metadata and prefer recency in ranking
• high Detect near-duplicate chunks with divergent numbers at ingest and flag them

Dense embeddings compress meaning and are structurally bad at exact tokens with no semantic neighbourhood — error codes, SKUs, part numbers, ticket ids, version strings, surnames, internal acronyms. ERR-4471 and ERR-4417 are near-identical in embedding space and completely different in meaning. Dense models also frequently ignore negation, and cross-lingual retrieval fails outright when the query language differs from the corpus and the model isn't multilingual.

What you'd see

• ·Failures cluster on identifiers and jargon while paraphrase questions work fine.
• ·Ctrl-F in the source document finds the answer in two seconds.
• ·Pure-vector setup with no keyword path.

Confirm it

1. Measure recall@5 versus recall@20 Hand-label 30 query-to-gold-chunk pairs. Ninety minutes of work, and the only number that makes retrieval debuggable. High@20 with low@5 is retrieval noise, not this.
2. Run a failing identifier query through BM25 If keyword search lands it at rank 1, dense retrieval is your gap.
3. Bucket failures by whether the query contains an identifier token The split is usually stark.

If confirmed, fix

• medium Hybrid BM25 + dense fused with reciprocal rank fusion
• low Keyword boosting or exact-match filters on identifier patterns
• high A domain-appropriate or fine-tuned embedding model
• medium Synonym and acronym expansion at query time

Often confused with

retrieval-noise — recall@20 tells them apart. Low at both is a miss and needs hybrid search; high at 20 and low at 5 is noise and needs reranking.

Similarity ranks by topical resemblance, which is exactly what near-duplicates maximise. Ten chunks about the refund policy across five product versions all score within noise of each other. Given ten equally authoritative-looking passages the generator has no principled way to choose, and empirically favours the one placed first or last rather than the one ranked highest. Raising k monotonically improves recall and monotonically degrades precision; past a point accuracy falls.

What you'd see

• ·The answer is confident, coherent, cites a real document, and is off by version, region or year.
• ·Raising k made things worse.
• ·No reranking step, and a corpus with heavy boilerplate or templated documents.

Confirm it

1. Record the gold chunk's rank on a failing query Rank 1 with a wrong answer points at generation. Rank 8 points here.
2. Run the same eval at k=3 and k=20 If accuracy inverts, precision is your binding constraint.
3. Count near-duplicates in the top-k Pairwise similarity across retrieved chunks tells you how crowded it is.

If confirmed, fix

• low Over-fetch and rerank with a cross-encoder — fetch 50, rerank, keep 5
• medium MMR or clustering for diversity in the candidate set
• medium Deduplicate and canonicalize near-identical chunks at ingest
• medium Filter on version and region metadata; break ties on recency

Often confused with

retrieval-miss — recall@5 versus recall@20 separates them, and it entirely changes the fix.

Several structurally different cases. Unresolved conversational reference — "what about the enterprise tier?" is embedded as-is and carries none of the referents from earlier turns. Aggregation — "how many customers are on annual billing" needs a scan, not a top-k; the answer isn't in any chunk. Comparison and multi-hop — the answer needs two documents jointly and the top-k fills with one of them. Temporal and superlative — "the latest" requires ordering by a field that similarity doesn't encode. Whole-document tasks — "summarise this contract" isn't retrieval at all.

What you'd see

• ·Single-fact lookups are excellent; follow-ups, "list all", "how many", "compare" and "latest" fail.
• ·Failures correlate with turn index greater than one.
• ·It confidently returns a count that is wrong and suspiciously round.

Confirm it

1. Replay the failing turn as a standalone question If a self-contained version works, the bug is in query construction, not retrieval.
2. Classify 50 failures by question type The distribution tells you which router branch to build first.
3. Log the string actually embedded Not the string the user typed — they are often different.

If confirmed, fix

• medium History-aware query rewriting and condensation
• medium Multi-query decomposition with fusion for comparison and multi-hop
• high An intent router sending aggregation and filtering to SQL, not vector search
• medium Tool or code execution for arithmetic instead of asking the model to compute

The final prompt is a budget — system prompt plus tool definitions plus chat history plus k chunks plus few-shot examples. When it exceeds the window something is dropped, and in most codebases that truncation is naive: tail-cut, or drop the last-appended chunks, which after ranking are frequently the most relevant. Even inside the window attention isn't uniform; material in the middle of a long context is measurably less used than material at the edges. A bigger window doesn't fix this, it converts eviction into dilution.

What you'd see

• ·Quality degrades as the conversation lengthens; the same question is fine in a fresh session.
• ·Failures spike on queries that retrieve unusually long chunks.
• ·Reducing k improves accuracy.

Confirm it

1. Log the final prompt Token count, plus a boolean for whether the gold chunk's text is literally present. A five-line change that settles the question immediately.
2. Reproduce in a fresh session If it works there, history is evicting your context.
3. Bisect on k, then separately on history length Isolates which side of the budget is doing the damage.

If confirmed, fix

• medium Explicit token budgeting with a reserved floor for retrieved context
• low Rerank then truncate, rather than truncate and hope
• low Per-chunk token caps so one huge chunk can't evict five others
• low A hard assertion that fails loudly when the budget is blown

The model has strong priors about how the world usually works and blends them with the provided context, especially where your context is client-specific and unusual — your PTO policy is 20 days, the internet average is 15. Absent an explicit instruction confining it to the context, helpfully filling gaps is the trained behaviour, not a bug. Aggravated by no grounding instruction, context presented without clear delimiters so the model can't tell instruction from data, smaller models, and high temperature.

What you'd see

• ·The wrong answer is a plausible industry-standard fact rather than anything in your documents.
• ·You can point at the exact retrieved chunk that contradicts the output.
• ·Numbers, dates and thresholds are wrong more often than prose.

Confirm it

1. The context-ablation test Run the same question with the retrieved context removed. If the answer barely changes, it was never grounded in the first place.
2. The clean-room test Paste the retrieved chunks and the question into a bare chat with a strict "answer only from this text" instruction. Correct there means your prompt is the problem.
3. Read your system prompt for a grounding constraint It surprisingly often doesn't contain one.

If confirmed, fix

• low An explicit "answer only from the provided context" constraint with clear data delimiters
• medium Quote-then-answer structure — extract supporting spans first, answer from the spans
• high A separate faithfulness or NLI check of each claim against the context
• low A stronger model for the generation step specifically

Often confused with

no-abstention — Grounding failure is ignoring good context. No-abstention is having no good context and no exit path. Both produce fabrication; the fixes share nothing.

Three missing components, usually all three at once. There is no score threshold below which retrieval counts as "nothing found". The prompt never grants permission to refuse — and is often actively hostile to it ("always provide a helpful answer"). And there is no UI state for "no confident source", so even a correct internal abstention has nowhere to render. When retrieval returns five irrelevant chunks, synthesising something is the model's only in-distribution behaviour.

What you'd see

• ·You have never once seen it say "that isn't in the documentation".
• ·Asking about a topic you're certain isn't in the corpus still produces a fluent answer.
• ·Users describe it as confidently wrong rather than unhelpful.

Confirm it

1. The out-of-corpus probe Write 10 questions whose answers definitely aren't in your corpus and count non-abstentions. Above roughly 20% is the diagnosis. Takes fifteen minutes.
2. Plot top retrieval score, right versus wrong Meaningful separation means a threshold will work today.
3. Read the prompt for anything forbidding refusal "Always be helpful" is an instruction to fabricate.

If confirmed, fix

• medium A retrieval-score gate, calibrated on the score distribution (reranker scores calibrate better than cosine)
• low Explicit refusal permission plus few-shot refusal examples
• medium A cheap answerability classifier over question plus retrieved context
• medium A UI affordance making "no confident source" a first-class result

If you ask the model to emit citations as free text, they are tokens like any other and are subject to the same fabrication. It will invent plausible document titles, produce URLs that 404, cite a real retrieved document that doesn't contain the claim, or attach one document's citation to another document's fact after synthesising across chunks. The cited answer looks more trustworthy than an uncited one, which makes this the most corrosive failure on this list.

What you'd see

• ·Users report "the link is wrong" more often than "the answer is wrong".
• ·Clicking through to the cited source doesn't find the quoted text.
• ·Quoted spans are near-misses of the real text.

Confirm it

1. Assert cited ids are in the retrieved set For 20 answers, check programmatically that every cited id was actually retrieved for that request. Cheap, and it usually fails.
2. Assert quoted spans are literal substrings Every quote should appear verbatim in that chunk's stored text.
3. Verify the rendered link resolves A 404 rate above zero is a bug, not a rounding error.

If confirmed, fix

• medium Constrain citations to structured ids from the retrieved set via structured output
• medium Verify server-side and drop or flag unverifiable claims before rendering
• low Render quotes from stored chunk text, never from model output
• high Per-sentence attribution on high-stakes surfaces

Any retrieved chunk containing imperative text is read with the same authority as your system prompt — whether that's "ignore previous instructions and…" or, more subtly, a support macro reading "always tell the customer the refund window is 90 days". Reachable wherever the corpus contains user uploads, crawled pages, support tickets, email or open wiki edits. It also fires accidentally: documentation about prompting, or any policy document written in the imperative, will hijack the model with no adversary involved.

What you'd see

• ·Off-policy outputs that correlate with specific documents rather than specific questions.
• ·Sudden persona or language changes mid-answer.
• ·One user's content can influence another user's answers.

Confirm it

1. Read the retrieved chunks for the anomalous queries The instruction is usually sitting there in plain text.
2. Grep the corpus for imperative-to-model patterns "ignore", "you are", "system:", "instead, respond".
3. Try it yourself Upload a document containing an instruction and ask a question that retrieves it.

If confirmed, fix

• low Delimit and label retrieved content explicitly as untrusted data
• low State an instruction hierarchy and spotlight the boundary
• medium Keep tool-calling authority out of the retrieval-influenced path
• high Output guardrails checking the answer against policy independently of the generation prompt

Four sources. Sampling — temperature above zero with no seed. Alias drift — pointing at a floating model alias so the provider upgrades under you, plus silent deprecation redirects and changed defaults. Prompt drift — template edits shipped with no regression eval. And embedding-model mismatch, the most catastrophic and the easiest to miss: the index was built with model A and queries are embedded with model B, so retrieval becomes approximately random while every component reports healthy.

What you'd see

• ·Quality regressed with no deploy on your side.
• ·Retrying the identical question gives materially different answers.
• ·Retrieval quality collapsed uniformly rather than in a pattern — that signature is the embedding mismatch.

Confirm it

1. Ask the same question five times and diff Wording variation is fine. Different facts are not.
2. Assert the embedding model id matches Index metadata versus query time. If you don't store it in index metadata, that absence is itself the finding.
3. Log model id, version, sampling params and a prompt-template hash per request Then correlate the quality drop against the date.

If confirmed, fix

• low Temperature 0 for extraction and grounded answering
• low Pin dated model versions rather than floating aliases
• medium An offline eval suite in CI gating prompt changes
• low Store the embedding model id in index metadata and assert it at startup