Naive embedding retrieval demos beautifully and disappoints in production. The failures are predictable, and mostly fixable without changing models.
Semantic search is excellent at finding text that means something similar and unreliable at finding text containing a specific identifier. Ask a pure vector system about an exact error code or product SKU and it will confidently return passages that are thematically adjacent and factually useless.
What we add before reaching for a bigger model
- Hybrid retrieval — combine keyword scoring with embeddings so exact identifiers are actually findable
- Chunk on document structure rather than fixed token counts; a chunk that splits mid-table helps nobody
- Keep titles and headings in the chunk text, so retrieval has the context a human would use
- Re-rank the top candidates before they reach the prompt — retrieving twenty and passing the best five beats retrieving five
- Always cite sources back to the user; it converts a silent hallucination into a checkable claim
The evaluation gap
Most RAG systems have no measurement, so quality is assessed by whoever demoed it last. Build a set of question-and-expected-source pairs early, even a small one, and measure retrieval separately from generation. When answers go wrong it is usually retrieval, and without that split you will spend weeks tuning prompts to fix a search problem.