How to use this checklist: Work through each section sequentially as you design your RAG system. Check off items as you complete them. Not every item applies to every use case—use your judgment based on your requirements.

1 Planning & Requirements

Define success metrics - What does "good" look like? (e.g., answer accuracy, response time, user satisfaction) Identify data sources - What documents/data will you index? How often do they update? Estimate scale - How many documents? How many queries per day? Storage requirements? Determine latency requirements - How fast do responses need to be? (p95, p99) Assess security/compliance needs - Data residency, PII handling, access controls

2 Document Processing & Chunking

Choose chunking strategy - Fixed-size, semantic, or document-structure-aware? Set chunk size - Test 256, 512, 1024 tokens. Balance context vs. precision. Implement chunk overlap - 10-20% overlap prevents context splitting mid-concept Preserve document structure - Maintain headers, sections, metadata in chunk context Handle special content - Tables, code blocks, lists—chunk appropriately for content type Extract metadata - Source, date, author, category, etc. for filtering

3 Embedding Model Selection

Evaluate embedding models - Test OpenAI, Cohere, Voyage, open-source options Consider domain specificity - General vs. domain-specific embeddings (legal, medical, code) Check dimension size - Higher dimensions = better accuracy but more storage/compute Calculate embedding costs - API costs for initial indexing + ongoing updates Test retrieval quality - Run test queries, measure recall@k on representative data

4 Vector Database Setup

Select vector database - Pinecone (managed), Weaviate (flexible), ChromaDB (local), etc. Choose similarity metric - Cosine, dot product, or Euclidean distance Configure indexing parameters - HNSW parameters, number of shards, replicas Set up metadata filtering - Define filterable fields for targeted retrieval Implement backup/recovery - Regular backups, disaster recovery plan Plan for scale - Understand limits, scaling strategy, cost projections

5 Retrieval Strategy

Set retrieval parameters - How many chunks to retrieve? (k=3-10 typical) Implement query transformation - Rephrase user queries for better retrieval Consider hybrid search - Combine vector similarity with keyword/BM25 search Add re-ranking - Use Cohere Rerank or similar to improve result order Filter by metadata - Narrow results by date, category, permissions, etc. Handle no-results cases - Fallback behavior when no good matches found

6 LLM Integration & Prompt Engineering

Select LLM provider - ChatGPT, Claude, Azure OpenAI, AWS Bedrock Design system prompt - Define role, behavior, how to use retrieved context Format context injection - How to present retrieved chunks in the prompt Add citation instructions - Require model to cite sources, include chunk IDs Handle context limits - Strategy when retrieved content exceeds token limit Set temperature/parameters - Lower temperature (0.1-0.3) for factual RAG responses

7 Evaluation & Quality Assurance

Create test query set - 50-100 representative questions with expected answers Measure retrieval accuracy - Are the right chunks being retrieved? (recall@k) Evaluate answer quality - Accuracy, completeness, relevance, citation quality Test edge cases - Ambiguous queries, no relevant data, conflicting information Set up automated evaluation - LLM-as-judge or RAGAS for continuous monitoring

8 Production Deployment

Implement caching - Cache embeddings, frequent queries, LLM responses Add rate limiting - Protect against abuse, manage costs Set up monitoring - Latency, error rates, cost tracking, quality metrics Configure logging - Log queries, retrievals, responses for debugging and improvement Implement error handling - Graceful degradation, user-friendly error messages Plan data updates - How/when to re-index documents, handle deletions Set up alerting - Get notified of failures, performance degradation, cost spikes

9 Continuous Improvement

Collect user feedback - Thumbs up/down, explicit feedback forms Analyze failure cases - Review low-rated responses, identify patterns Monitor retrieval drift - Does retrieval quality degrade as data grows? A/B test improvements - Test chunking strategies, prompts, retrieval params Update documentation - Maintain architecture docs, runbooks, troubleshooting guides

RAG Architecture Checklist