RAG Is a Retrieval Problem Before It Is an LLM Problem

Sample article — starter content for NeuralSys.

Introduction

When RAG answers are wrong, teams usually blame the LLM and upgrade the model. In my experience the generator is guilty maybe 20% of the time. The other 80% is retrieval: bad chunks, bad ranking, or missing content entirely.

Diagnose Before You Tune

Run this checklist on 30 failing queries before changing anything:

1. Was the gold document retrieved at all? (recall@k)
2. Was it ranked in the top 3? (ranking)
3. Was it chunked so the answer survived? (chunking)
4. Did the generator ignore good evidence? (generation — only now blame the LLM)

def diagnose(query, gold_doc_id, k=10):
    hits = retriever.search(query, k=k)
    ids = [h.doc_id for h in hits]
    return {
        "recalled": gold_doc_id in ids,
        "rank": ids.index(gold_doc_id) if gold_doc_id in ids else None,
        "chunk_preview": hits[0].text[:300] if hits else None,
    }

Chunking Is a Design Decision

There is no universal chunk size. Match chunks to questions:

Hybrid Retrieval Wins

Pure dense search misses exact terms (error codes, names, SKUs). Pure BM25 misses paraphrase. Combine them:

Key Takeaways

• Measure recall@k before touching the generator.
• Chunk for the questions, not for the embedding model.
• Hybrid retrieval + reranking is the default serious baseline.