Why Your RAG System Returns Garbage (and Where to Look)

A retrieval-augmented generation system gives a confident wrong answer, and the team’s first move is to rewrite the prompt. That rarely helps, because the prompt is the last link in the chain. If the right information never made it into the context window, no wording will bring it back.

RAG fails in a predictable order, and debugging in that order saves weeks. This is where to look, from the top.

First: did retrieval find the right chunk?

Before touching anything else, answer one question for a query that is producing a bad answer: is the correct information in the retrieved chunks at all?

Log the chunks your retriever returns for a failing query and read them. That single step settles most RAG debugging, because it splits the problem in two. Either the right context was retrieved and the model mishandled it, which is a generation problem, or the right context was never retrieved, which is a retrieval problem. They have different fixes, and you cannot tell which one you have without looking.

When the right chunk is missing, the prompt was never the issue. Teams that skip this step tend to spend days on prompts that could not have helped.

The retrieval problems, roughly by how often they bite

When the right chunk is not being found, the cause is usually one of these.

Chunking that split the answer. The passage that answers the question was cut across two chunks, so neither matches well on its own. This is the most common and most overlooked cause. Fixed-size splitting by token count cuts through the middle of the very passages that hold complete answers. Splitting on headings, paragraphs or sentences, with a modest overlap, keeps them together.

# Fixed-size: can cut an answer in half
chunks = [text[i:i + 800] for i in range(0, len(text), 800)]

# Structure-aware: split on headings, then pack paragraphs up to a size limit
sections = split_on_headings(text)
chunks = [c for s in sections for c in pack_paragraphs(s, max_tokens=400, overlap=1)]

A mismatch between query and document language. The user asks in plain words and the document is written in formal or technical terms, so semantic similarity alone misses the link. Hybrid search, which combines keyword matching such as BM25 with vector similarity, addresses this. It also helps with exact terms like product codes and error messages, which embeddings often blur.

Too few chunks retrieved. Pulling the top 3 when the answer needed context from the top 8 is cheap to test. Raise the count and see whether quality jumps.

No reranking. Vector search returns roughly relevant chunks, and the best one may sit in fourth place, outside your cutoff. A reranking pass over the top 20 or 30 candidates fixes the ordering for a small cost in latency.

Stale or missing documents. Sometimes the answer simply is not in the index. Check that the source document was ingested, that the latest version replaced the old one, and that nothing was dropped by a parser that mangled a table or a PDF.

None of these are prompt problems. They are all about what got into the window.

Then, and only then, generation

If the right context is being retrieved and the answer is still wrong, the generation layer deserves attention.

• The model has the right chunk and ignores it in favor of what it learned in training. Instruct it to answer only from the provided context, and to say it does not know when the context does not cover the question.
• The right chunk is present but buried among irrelevant ones, which dilutes the signal. That argues for retrieving fewer, better chunks.
• The model pads or hedges. That is a real prompt fix, and at this point in the chain it is the correct one.

Ask for citations to the retrieved chunks as part of the answer. They make wrong answers easier to diagnose, because you can see which chunk the model used.

Build the eval before you tune

RAG debugging spirals when changes are judged by feel on a handful of queries. An improvement on three examples can hide a regression on thirty others you did not check.

Assemble a set of real questions with known good answers. Twenty is enough to start. For each, record which document or section should be retrieved, so you can measure retrieval separately from the final answer. Run every change against the set. When you adjust chunking, you will see right away whether retrieval got better or only different. Track two numbers: how often the correct chunk appears in the retrieved set, and how often the final answer is correct. If the first number is low, fix retrieval. If the first is high and the second is low, fix generation.

Without this, you are optimizing blind and will chase the same problems indefinitely.

The order that saves weeks

1. Log and read the retrieved chunks for failing queries.
2. If the right chunk is missing, fix chunking, add hybrid search, retrieve more, rerank, and confirm the document is indexed.
3. If the right chunk is present and the answer is still wrong, fix generation.
4. Hold every change to a fixed eval set so you can tell improvement from noise.

Almost every “our RAG does not work” problem resolves in steps one and two. The prompt, which is where everyone starts, is usually the last thing that needed changing.

If your RAG system is returning answers you cannot trust and you are not sure where it breaks down, that is the kind of problem Twill Software diagnoses with teams. The first consultation is free: get in touch.