AI Engineering 10 min read RAG / Retrieval

Understanding RAG by Building One

Retrieval-augmented generation makes much more sense once you stop treating it as a diagram and actually build the pipeline.

Documents Chunks Embeddings

RAG

Vector DB Retrieval LLM

The short version

RAG is usually explained as a clean sequence: retrieve relevant information, place it into a prompt, and let a language model generate an answer.

That explanation is correct, but it hides most of the decisions that determine whether the system is actually useful.

Once I started building one, the interesting questions moved upstream: how documents are split, what becomes retrievable, how relevance is measured, how much context is passed forward, and how the final answer is evaluated.

RAG is not one model. It is a pipeline.

A retrieval-augmented system combines two separate capabilities: finding useful information and generating a response from that information.

The language model is only the final stage. Before the model ever sees a question, documents must be prepared, divided into retrievable units, represented numerically, and stored somewhere that supports similarity search.

Mental model Retrieval decides what evidence the model gets to see. Generation decides what the model does with that evidence.

That distinction changed how I approached debugging. If an answer is weak, the first question is no longer simply “Why did the model get this wrong?” It becomes “Did the model receive the right evidence in the first place?”

Index first. Retrieve later.

A practical RAG system is easier to understand when I separate it into two pipelines: an indexing pipeline and a query pipeline.

INDEXING

01 Documents

→

02 Chunks

→

03 Embeddings

→

04 Vector store

QUERY

01 Question

→

02 Retrieve

→

03 Context

→

04 Answer

The indexing pipeline determines what can later be found. The query pipeline determines which of those indexed units are selected for a specific question.

Chunking changes what becomes retrievable.

Chunking initially looked like a preprocessing detail. It turned out to be an architectural decision.

A retrieval system does not search a document in the abstract. It searches the units I created when I split that document. Those units become the system’s searchable memory.

Too small Precision may improve, but context can disappear.

Too large Context is preserved, but retrieval can become noisy.

The right size depends on document structure, question type, and how much surrounding context is needed to make a passage meaningful.

Similarity is useful. Relevance is the real goal.

Embeddings transform text into numerical representations that make semantic comparison possible. That allows a user question to be compared with stored chunks even when the wording is different.

But a mathematically similar result is not automatically the most useful evidence for the question being asked.

What I’m testing

Retrieval quality depends on more than the embedding model: chunk boundaries, document structure, query wording, the number of retrieved passages, and filtering strategy all influence the result.

This is where RAG begins to feel less like “attach a vector database” and more like an information-retrieval problem.

More context is not always better context.

Once retrieval returns candidate passages, another decision appears: how much of that material should actually be passed into the model?

My initial instinct was that more retrieved text would give the model more information and therefore improve the answer. In practice, irrelevant context can dilute the useful evidence.

A larger context window does not remove the need to decide what belongs in the context.

The context-building stage therefore needs its own discipline: select relevant evidence, preserve enough surrounding meaning, and avoid overwhelming the prompt with unrelated passages.

Fluent is not the same as grounded.

The language model can produce a convincing answer even when the retrieved evidence is weak, incomplete, or unrelated.

That makes grounding one of the central requirements of a useful document assistant. The answer should be explainable from the supplied evidence.

Evidence What did retrieval actually return?

→

Instruction How strictly should the model use it?

→

Answer Can the claims be supported?

Prompt design helps, but prompt design cannot repair missing evidence. Grounding begins with retrieval.

“Looks good” is not an evaluation strategy.

One of the biggest changes in my thinking was recognizing that evaluating only the final answer hides where the system failed.

A useful evaluation approach should inspect retrieval and generation separately.

01 Retrieval relevance

02 Grounding

03 Completeness

04 Unsupported claims

05 Consistency

06 No-answer behavior

This makes experimentation more useful because I can change one variable and observe whether retrieval, grounding, or answer quality actually improves.

Most failures leave clues upstream.

Retrieval miss

The correct information exists but never reaches the prompt.

Partial retrieval

The system finds some evidence but misses another required piece.

Context pollution

Irrelevant passages compete with the useful evidence.

Unsupported generation

The answer goes beyond what the retrieved context supports.

No-answer failure

The model answers even though the document collection cannot.

The model stopped being the center of the system.

Before building the pipeline, I thought about RAG primarily as a way to give an LLM access to private information.

After working through the individual stages, I now think of it as a small information system where the model is only one component.

Before “Give the LLM my documents.”

→

Now “Build a retrieval system that gives the model the right evidence.”

That shift makes debugging more concrete. It also makes system quality less mysterious.

Turn the mental model into measurements.

The next step is not to add more features. It is to compare the decisions already inside the pipeline.

Compare chunk sizes Test overlap strategies Compare retrieval quality Build an evaluation set Track failure patterns

That work continues in the AI Lab, where the RAG Knowledge Assistant is the current experiment.

Follow the experiment →

Keep Building

Understanding came from making the pipeline real.

The diagram was useful. Building it exposed the decisions the diagram leaves out.

View the Experiment → Back to Insights