2025-06-15
Most RAG projects look great in a Jupyter notebook. Then someone asks: "Can we connect it to our SharePoint?" and everything falls apart.
After building RAG systems for SaaS companies, law firms, and healthcare networks, we've seen the same failure patterns repeatedly. Here are the five that kill projects - and what to do instead.
1. Chunking Strategy Was an Afterthought
The failure: Dumping documents into fixed 512-token chunks without understanding document structure.
The fix: Structure-aware chunking. Respect headings, tables, and metadata. Use parent-child chunking for long documents. Test retrieval quality per document type before scaling ingestion.
2. No Evaluation Framework
The failure: "It seems to work" is not a QA strategy.
The fix: Build a golden dataset of 50–100 question-answer pairs from real user queries. Measure precision@k, answer faithfulness, and citation accuracy. Run evals on every pipeline change.
3. Ignoring Permissions
The failure: Retrieving documents the user shouldn't see.
The fix: Filter at retrieval time using the same permission model as your source system. Never rely on post-generation filtering alone.
4. Single-Stage Retrieval
The failure: One embedding search returns irrelevant chunks for complex queries.
The fix: Hybrid search (dense + BM25), query expansion, reranking with a cross-encoder, and metadata filters. The extra latency is worth the accuracy gain.
5. No Cost Controls
The failure: Sending entire document chunks to the LLM on every query.
The fix: Caching, smaller context windows, model routing (cheap model for simple queries), and token budgets per request.
Need help shipping RAG to production? Book a discovery call with Sanidhyam AI Labs.
Ready to ship RAG to production?