Skip to main content

Production RAG with Spring AI

Building a Retrieval-Augmented Generation (RAG) prototype is straightforward: add a few dependencies, index some documents, and start asking questions. Moving that same system into production—where it must serve hundreds of users, handle sensitive data, and operate with high availability—is an order of magnitude more difficult. Hallucinations, stale knowledge, latency spikes, security vulnerabilities, and runaway costs are real threats that demand a disciplined engineering approach.

Spring AI provides the architectural foundation to tackle these challenges. Its portable abstractions, tight integration with the Spring ecosystem (security, observability, messaging), and production‑grade features (retry, structured output, advisor chains) make it an ideal platform for enterprise RAG. This guide covers the end‑to‑end design, optimization, and operation of production RAG systems with Spring AI.

Production RAG Architecture​

A production‑grade RAG system extends the basic retrieval pipeline with robust infrastructure for ingestion, monitoring, caching, and security.

  • Spring Boot Application: orchestrates the RAG flow; enforces security and rate limiting.
  • Spring AI Framework: provides ChatClient, advisors, and model abstractions.
  • EmbeddingModel: converts queries into vectors; should be consistent with the one used for indexing.
  • VectorStore / Vector Database: stores and retrieves document embeddings. Production setups use dedicated databases (PGVector, Milvus, Pinecone, etc.).
  • Retrieval & Prompt Construction: Advisors handle retrieval, re‑ranking, and prompt augmentation.
  • LLM: the generative model; may be a cloud provider or a self‑hosted instance.
  • Supporting Infrastructure: handles asynchronous ingestion, caching, monitoring, authentication, and document storage.

Core Components of a Production RAG System​

Each component must be designed for reliability, scalability, and maintainability:

ComponentDescriptionProduction Considerations
Document IngestionReading files from S3, databases, CMSHandle rate limits, failures, and incremental sync
Preprocessing & ChunkingCleaning, splitting into chunksChoose chunking strategy based on document type; preserve metadata
Embedding GenerationConverting chunks to vectorsBatch for efficiency; cache stable content; manage model versions
Metadata EnrichmentAdding source, access control, timestampsEssential for filtering and governance
Vector StoragePersisting embeddingsSelect database based on scale, latency, and cost requirements
RetrievalSemantic search with filtersTune top‑K, similarity thresholds, and hybrid search
Prompt AssemblyBuilding context for the LLMControl token usage, include citations, guard against prompt injection
LLM InferenceGenerating answersImplement retries, fallbacks, and circuit breakers
Response ValidationChecking for hallucinations or off‑topic answersApply post‑generation filters or require source grounding

These components form a pipeline that must be monitored and optimized continuously.

Document Processing Pipeline​

A robust ingestion pipeline is the foundation of a good RAG system.

  • Format Handling: Use Apache Tika or dedicated readers for PDFs, Word documents, HTML, and Markdown. OCR (Tesseract, cloud OCR) is necessary for scanned images.
  • Cleaning: Remove headers, footers, navigation elements, and boilerplate.
  • Chunking: Apply token‑based, recursive, or structure‑aware splitting to balance context and precision.
  • Metadata: Extract from file properties (source, author, date) and content (title, department). Include access‑control tags.
  • Incremental Sync: Detect new, modified, and deleted documents; update the index without full re‑indexing.

Spring AI’s DocumentReader and DocumentTransformer interfaces provide hooks for building this pipeline. For high‑volume systems, decouple ingestion using a message queue (Kafka, RabbitMQ) and parallel workers.

Embedding Strategy​

Embeddings power semantic retrieval. In production, consistency and efficiency are paramount.

  • Model Selection: Choose an embedding model that matches your domain (code, legal, multilingual). OpenAI’s text-embedding-3-small offers a good balance of cost and quality; open‑source models (BGE, E5) allow self‑hosting.
  • Consistency: The same model (and same dimension) must be used for both indexing and querying. Changing the model requires a full re‑index.
  • Version Management: Store the embedding model name and version alongside vectors. This facilitates planned migrations.
  • Batching: Use EmbeddingModel.embed(List<String>) to reduce API overhead. Many providers have token‑per‑minute limits; implement queuing and rate limiting.
  • Caching: Cache embeddings for stable content (e.g., archived documents) using a key‑value store (Redis, Caffeine). A content hash serves as the cache key.

Vector Database Design​

The vector database is the heart of retrieval. Production setups demand careful schema and index design.

// Example: indexing a document with metadata
Document doc = new Document("Spring AI provides ChatClient for streaming responses.");
doc.getMetadata().put("source", "spring-ai-reference.pdf");
doc.getMetadata().put("version", "1.4.0");
doc.getMetadata().put("accessLevel", "public");
vectorStore.add(List.of(doc));

Best practices:

  • Namespaces / Collections: logically separate content (e.g., docs.engineering, docs.hr) to improve performance and governance.
  • Index Type: Use HNSW for high recall, IVFFlat for write‑heavy workloads. Tune efConstruction, M, and efSearch.
  • Metadata Filtering: Store fields used in filters as indexed metadata. Keep filter fields minimal to reduce overhead.
  • Partitioning: For multi‑tenancy, include tenantId and enforce it in every query. Some databases support partition‑based isolation.
  • Hybrid Search: Combine vector search with keyword (BM25) if exact matches are important. Use stores that support both (e.g., OpenSearch, Weaviate) or implement fusion.

Database choice depends on scale, latency, and operations budget (see Vector Databases).

Retrieval Optimization​

The retrieval stage directly impacts answer quality. Optimize it relentlessly.

  • Similarity Search: Use SearchRequest to set topK, similarityThreshold, and filterExpression. A typical starting point: retrieve 20‑50 candidates, then trim.
  • Metadata Filtering: Apply filters early to reduce the search space. Example: "department == 'Engineering' && year >= 2025".
  • Hybrid Retrieval: Run vector and keyword searches in parallel, then fuse results (Reciprocal Rank Fusion).
  • Re‑ranking: Use a cross‑encoder (e.g., Cohere Rerank, BGE‑Reranker) to reorder the fused candidate list and keep only the top‑5 most relevant.
  • Contextual Retrieval: For ambiguous queries, use query rewriting or expansion before retrieval.

Retrieval quality can be monitored with metrics like recall@k, MRR, and NDCG, using a curated evaluation dataset.

Prompt Engineering for Production​

A well‑crafted prompt ensures the LLM uses the retrieved context correctly without hallucinating.

String systemMessage = """
    You are a helpful assistant. Use ONLY the following context to answer the question.
    If you cannot answer based on the context, say "I don't have that information."
    Cite the source documents when possible.
    """;
String userMessage = "Context:\n" + retrievedContext + "\n\nQuestion: " + userQuery;

String answer = ChatClient.create(chatModel)
    .prompt()
    .system(systemMessage)
    .user(userMessage)
    .call()
    .content();

Key considerations:

  • Token Budgeting: Monitor token usage of retrieved context. Trim or summarise if it exceeds the model’s context window.
  • Citation: Include source metadata in the prompt and instruct the model to cite it; this improves trust and debuggability.
  • Prompt Templates: Externalize templates (e.g., PromptTemplate) for A/B testing and version control.
  • Conversation Memory: For multi‑turn interactions, integrate ChatMemory to include chat history without overwhelming the context.

Scalability​

A production RAG service must handle growing document volumes and query traffic.

  • Horizontal Scaling: Deploy stateless Spring AI instances behind a load balancer. The vector database and embedding service should scale independently.
  • Async Ingestion: Use a queue to decouple document updates from the request path; workers handle chunking, embedding, and indexing.
  • Caching: Embedding cache (Redis) and possibly LLM response cache (for identical queries) drastically reduce cost and latency.
  • Autoscaling: Leverage Kubernetes HPA (Horizontal Pod Autoscaler) based on CPU/memory or custom metrics (e.g., search latency).
  • CDN / Object Storage: Serve static embeddings or pre‑generated answers via CDN; store original documents in S3/Azure Blob.

Security​

Enterprise AI must protect sensitive data and prevent misuse.

  • Authentication & Authorization: Integrate with Spring Security, OAuth2, or API keys. Each request must carry an identity.
  • RBAC / Tenant Isolation: Ensure users can only retrieve documents they’re authorized to see. Enforce tenantId or accessLevel filters in every query.
  • Prompt Injection Defense: Sanitize user input. Use system prompts that constrain the model’s behavior, and consider a guardrail advisor that scans inputs for malicious patterns.
  • Sensitive Data Masking: Redact PII before sending data to external LLMs, or use self‑hosted models to keep data in‑house.
  • Encrypted Storage: Encrypt vector stores at rest and in transit. Use VPC peering or private endpoints.
  • Audit Logging: Record all retrieval and generation events (user, query, retrieved documents, response) for compliance and debugging.

Observability​

You cannot improve what you can’t measure. Instrument every stage of the RAG pipeline.

  • Latency Tracking: Measure retrieval (vector search + re‑ranking), augmentation, and generation times. Set SLIs and SLOs.
  • Token Usage: Monitor prompt and completion tokens per request, aggregated by user or department. Use this for cost attribution.
  • Retrieval Quality: Log the similarity scores and ranked positions of retrieved documents. Compare against a golden dataset periodically.
  • Metrics & Traces: Use Micrometer and OpenTelemetry to export metrics (Prometheus) and traces (Jaeger, Zipkin). Auto‑configuration in Spring AI simplifies this.
  • Dashboards: Build Grafana dashboards showing request rate, error rate, p95 latency, and token consumption.

Cost Optimization​

LLM and embedding calls are the primary cost drivers. Reduce them systematically.

  • Embedding Cost: Batch insertions, cache stable content, and pre‑compute embeddings for static documents.
  • Retrieval Tuning: Reduce the number of retrieved candidates and apply stricter similarity thresholds.
  • Prompt Compression: Summarise retrieved chunks before inclusion, or use models with larger context windows to avoid splitting.
  • Model Selection: Use smaller, cheaper models for simple queries; reserve large models for complex reasoning. Consider self‑hosted Llama3 or Qwen models via Ollama.
  • Caching: LLM responses for identical or highly similar queries can be cached with a cache‑aside pattern.

Reliability​

Failures are inevitable. Design for graceful degradation.

@Bean
public ChatClient chatClient(ChatModel primary, ChatModel fallback) {
    return ChatClient.builder()
        .defaultChatModel(primary)
        .defaultAdvisors(
            new RetryAdvisor(3, Duration.ofSeconds(1)),
            new FallbackAdvisor(fallback)
        )
        .build();
}
  • Retries: Apply exponential backoff for transient provider errors using RetryAdvisor or Spring Retry.
  • Circuit Breakers: Use Resilience4j to stop calling a failing provider after a threshold.
  • Fallback Models: Route requests to a secondary model (e.g., local Ollama) when the primary is down.
  • Graceful Degradation: If retrieval fails, return a cached answer or a polite "service temporarily unavailable" message.
  • Timeout Handling: Set timeouts on all external calls (embedding, vector search, LLM) to avoid thread exhaustion.

Testing Production RAG​

Testing a RAG system requires more than unit tests. Establish a comprehensive quality assurance strategy.

  • Unit Tests: Mock EmbeddingModel and VectorStore to test chunking, prompt assembly, and advisor logic in isolation.
  • Integration Tests: Use Testcontainers to spin up a real vector database (PGVector) and a mock LLM to test the entire pipeline.
  • Retrieval Evaluation: Create a dataset of queries and expected relevant document IDs. Run retrieval and measure recall@k, precision@k.
  • Response Evaluation: Use an evaluation framework (e.g., RAGAS) or manually score answers for faithfulness and relevance.
  • Regression Testing: Automate the evaluation suite and run it on every deployment to catch regressions in chunking, embeddings, or prompts.
  • Hallucination Testing: Craft adversarial queries that might induce hallucinations and verify that the system either answers “I don’t know” or remains grounded.
  • Load Testing: Simulate production traffic with JMeter or k6 to validate scaling and latency targets.

Common Production Mistakes​

  • Oversized Chunks: Large chunks bloat the context window and reduce relevance. Experiment to find the optimal size.
  • Poor Metadata: Missing or inconsistent metadata prevents effective filtering and hinders governance.
  • Outdated Embeddings: Failing to re‑index when documents change leads to stale answers.
  • Missing Monitoring: Without observability, performance degradation and cost spikes go unnoticed.
  • Weak Security: Exposing the vector store or LLM without proper authentication can lead to data leaks.
  • Excessive Token Usage: Sending too many retrieved chunks to the LLM increases cost and reduces answer quality.
  • Ignoring Evaluation: Subjective “it looks good” is not a valid quality metric; automated evaluation is essential.
  • Hardcoded Prompts: Storing prompts in code prevents rapid tuning and A/B testing.

Best Practices Checklist​

  • Architecture: Separate ingestion from retrieval; use async messaging.
  • Ingestion: Automate document sync; clean and normalize text before chunking.
  • Chunking: Align chunk size and overlap with document structure; evaluate retrieval impact.
  • Embeddings: Use a consistent model; cache stable embeddings; plan model versioning.
  • Vector Database: Choose based on scale; design metadata schema; implement hybrid search if needed.
  • Retrieval: Apply metadata filters; use re‑ranking; tune top‑K and thresholds.
  • Prompts: Externalize templates; enforce token budgets; include citation instructions.
  • Security: Enforce RBAC and tenant isolation; defend against prompt injection; encrypt data.
  • Observability: Monitor latency, token usage, and retrieval quality; set up dashboards and alerts.
  • Deployment: Use containers and Kubernetes; autoscale based on metrics; implement CI/CD.
  • Testing: Automate unit, integration, and evaluation tests; run continuous accuracy benchmarks.
  • Maintenance: Regularly update embeddings; review and refresh prompts; prune stale documents.

Conclusion​

Production RAG is a multidisciplinary engineering challenge that extends far beyond hooking up a vector database to an LLM. It requires careful design of ingestion pipelines, embedding consistency, retrieval optimization, security, and observability—all while controlling cost and ensuring reliability. Spring AI provides a robust, portable foundation that integrates seamlessly with the Spring ecosystem, allowing you to build enterprise‑grade RAG systems that scale, stay secure, and deliver trustworthy answers.

By applying the patterns and practices outlined in this guide, you can move from a fragile prototype to a hardened production service. Continue your journey with the deeper dives in the RAG section and the enterprise patterns in the Enterprise AI section.