Better chunks
Meaningful boundaries should improve retrieval relevance.
RAG Knowledge Assistant
Building a retrieval-augmented assistant over private documents while testing how chunking, embeddings, retrieval quality, prompting, grounding, and evaluation affect the usefulness of the final answer.
RAG
Embeddings
Vector Search
Prompting
Evaluation
Documents
Chunks
Embeddings
→
RAG
Retrieve + Generate
→
Retrieve
Context
Answer
QUESTION
Can I build a useful assistant
over private documents?
01
The Question
What actually makes a RAG system useful?
The system can return an answer very quickly. That does not mean the answer is relevant, grounded, complete, or trustworthy.
The experiment is designed to move past the architecture diagram and test the individual decisions that influence retrieval and generation quality.
EXPERIMENT QUESTION
How do chunking, embeddings, retrieval quality,
prompt construction, and evaluation affect the usefulness
of a private-document assistant?
02
Working Hypothesis
Retrieval quality matters before generation.
My current assumption is that many apparent LLM failures are actually retrieval failures upstream.
Better retrieval
Relevant context should improve answer grounding.
Better prompts
Clear instructions should reduce unsupported answers.
Better evaluation
Defined criteria should make improvements measurable.
03
System Design
Two pipelines. One answer.
RAG separates document preparation from question-time retrieval. Understanding both paths is essential.
INDEXING
01
Documents
Private source files
→
02
Chunk
Split into passages
→
03
Embed
Create vectors
→
04
Store
Vector database
QUERY
01
Question
User query
→
02
Retrieve
Relevant chunks
→
03
Context
Grounding material
→
04
Generate
Final answer
04
Experiment Variables
Change one thing. Observe what moves.
Too small may lose context. Too large may reduce retrieval precision.
Overlap may preserve continuity but increases duplication.
Different embedding approaches may affect semantic retrieval.
More context is not automatically better context.
Instructions influence how strictly the model uses context.
Structured, narrative, and mixed documents may require different handling.
05
Evaluation
Useful needs a definition.
A convincing answer can still be wrong. Evaluation needs criteria beyond how fluent the response sounds.
01
Retrieval relevance
Did the system retrieve passages that actually address the question?
02
Grounding
Can the answer be supported by the retrieved material?
03
Completeness
Did the answer cover the important parts of the question?
04
Unsupported claims
Did the model introduce information that was not present in context?
05
Retrieval consistency
Do similar questions retrieve consistently useful evidence?
06
Failure transparency
Does the assistant recognize when the documents do not contain an answer?
06
Current Observations
The interesting part is usually upstream.
Retrieval failures often look like model failures.
If the relevant information never reaches the prompt, generation cannot recover it reliably.
More retrieved text can reduce clarity.
Increasing context size can introduce irrelevant material and make grounding harder rather than easier.
Chunk boundaries change what becomes retrievable.
Splitting content is not merely preprocessing. It changes the units the system can reason over.
07
Failure Modes
What breaks first?
Retrieval Miss
Relevant information exists but is not retrieved.
Partial Retrieval
Only part of the evidence needed for the answer is found.
Context Pollution
Irrelevant passages dilute the useful evidence.
Unsupported Answer
Generation goes beyond what the retrieved evidence supports.
False Confidence
The response sounds certain even when the evidence is weak.
No-answer Handling
The system should recognize when the collection cannot answer.
08
Next Tests
Turn assumptions into evidence.
Experiment Status
Still building. Still testing.
The purpose of this experiment is not to produce one perfect RAG demo. It is to understand which design choices actually improve retrieval and answer quality.