How I Approach Production RAG Architecture
The retrieval, security, evaluation, latency and observability decisions that matter after the proof of concept.
A production RAG system is a distributed application with retrieval quality, security, latency, observability and cost requirements. The LLM is only one component.
Treat ingestion and retrieval as owned capabilities
Document parsing, chunking, metadata, embeddings, access controls and index lifecycle need explicit design. Retrieval quality should be measured with representative queries rather than judged only from a few demos.
Keep orchestration modular
Separate business workflows, retrieval, prompt/context assembly and model-provider integration. That makes it easier to change models, vector stores or ranking strategies without rewriting the whole application.
Design for security and traceability
Apply document-level authorization before retrieval, minimize sensitive context, redact where appropriate and log the decisions needed for debugging without exposing private content. Tool or function calls should be allow-listed and validated.
Measure quality, latency and cost together
Capture retrieval hit quality, answer quality, refusal/failure rates, end-to-end latency, token usage and user feedback. Production optimization is a trade-off between better context, faster answers and sustainable cost.