AI LAB / RAG / Local AI
RAG Retrieval on Local Models
Comparing retrieval quality and latency across compact local embedding models.
RAGLocal AIBenchmarks
- EXPERIMENT
- EXP-014
- STATUS
- complete
- DATE
- Aug 26, 2026
- TAGS
- RAG · Local AI · Benchmarks
EXPERIMENT / HYPOTHESIS
A compact local embedding model can retain at least 90% recall while avoiding external inference latency.
- 01Build a 500-query dataset across six document domains
- 02Index identical chunks with three embedding models
- 03Measure recall@5, MRR, indexing time, and query latency
- Recall@5
- 91.4%
- P95 latency
- 312ms
- Queries
- 500
CONCLUSIONThe best compact model cleared the recall threshold and was faster for private, interactive workloads, but struggled more with jargon-heavy queries.
Setup
Every model received the same normalized chunks and queries. Results were evaluated against human-labeled relevant passages.
- Fixed chunking
- Warm and cold latency
- Domain-level slices
Result
Local retrieval was competitive for general queries. Hosted embeddings still led on specialized terminology, suggesting a workload-aware routing policy.
- Strong general recall
- Predictable privacy
- Domain-specific quality gap