PROJECT / RAG / Evaluation
Document QA System
A retrieval system built to make document answers traceable, measurable, and fast enough to use.
RAGEvaluationRetrieval
- STATUS
- shipped
- YEAR
- 2026
- ROLE
- Backend & AI Engineer
- STACK
- Python · FastAPI · PostgreSQL · pgvector
SYSTEM / ARCHITECTURE
How it fits together
- 01Document pipeline
- 02Hybrid retriever
- 03Reranker
- 04Citation builder
- 05Evaluation service
- Recall@5
- 93.2%
- P95 latency
- 820ms
- Evaluated queries
- 500
The problem
Prototype question-answering felt impressive but could not show why an answer was trustworthy. The product needed citations, predictable latency, and a repeatable quality measure.
- Mixed document formats
- Ambiguous user vocabulary
- Answers must trace back to exact passages
The system
Documents are normalized and chunked with structural metadata. Lexical and vector results are merged, reranked, then passed to an answer step that can only cite retrieved passages.
- Hybrid retrieval
- Offline golden-query evaluation
- Per-stage latency and quality telemetry
Outcome
The shipped service made retrieval quality measurable and reduced the time needed to diagnose bad answers from hours to minutes.