Hybrid Search in Spring AI RAG: Vector + Keyword

Retrieval-augmented generation systems traditionally rely on vector similarity—embedding a query and finding documents whose vectors are closest in the semantic space. While powerful, this approach alone can miss the precision needed in enterprise environments, where exact terms, identifiers, and structured filters are critical. Hybrid search solves this by combining dense vector retrieval with sparse keyword retrieval and intelligent ranking, providing the best of both worlds.

Spring AI gives you the building blocks to implement hybrid search pipelines that leverage vector databases alongside traditional search engines or keyword indexes. This article explains why hybrid search matters, how it fits into the RAG architecture, and how to design production-grade retrieval that is both semantically rich and terminologically precise.

Why Vector Search Alone Is Not Enough

Vector search excels at understanding meaning: a query about “cheap smartphones with great cameras” can retrieve documents discussing “budget-friendly phones with excellent photography.” However, enterprise documents contain numerous elements where exact matching is vital:

• Exact technical terms – class names (ChatClient.Builder), API identifiers (VectorStore), or configuration keys (spring.ai.openai.api-key). A vector model may associate these with related concepts but fail to distinguish a specific method name.
• Numbers and codes – error codes, product IDs, version strings. Embeddings treat them as tokens without appreciating their unique identity.
• Compliance and regulatory documents – legal clauses, policy numbers, or standards references that must be matched verbatim.

Consider a query like “Spring AI ChatClient streaming configuration”:

• Vector search might retrieve documents about “asynchronous chat responses” or “streaming API concepts”, which are semantically relevant but may not contain the exact configuration parameter the user needs.
• Keyword search ensures the document actually includes “ChatClient”, “streaming”, and “configuration”.

Combining both yields results that are both contextually appropriate and terminologically correct.

What Is Hybrid Search?

Hybrid search fuses two distinct retrieval modalities:

1. Dense Retrieval – based on embeddings; captures semantic meaning.
2. Sparse Retrieval – based on keywords (e.g., BM25, inverted index); ensures exact term presence.

A ranking layer then merges and re‑scores the combined candidate set.

This architecture decouples retrieval strategies, allowing you to tune each independently and adapt the pipeline to different document types.

Hybrid Search Architecture in RAG

In a complete RAG flow, hybrid retrieval sits between query understanding and context injection:

User Question
      |
      v
Query Understanding
      |
      +----------------+
      |                |
      v                v
Embedding Search   Keyword Search
      |                |
      +----------------+
              |
              v
       Result Fusion
              |
              v
        Re-ranking
              |
              v
            LLM

• Query Understanding – may rewrite or expand the query to improve recall.
• Embedding Search – converts the query to a vector via EmbeddingModel and searches the VectorStore.
• Keyword Search – uses a term‑based engine (Lucene, Elasticsearch, OpenSearch) to find documents containing exact matches.
• Result Fusion – normalizes scores, deduplicates, and merges candidates.
• Re‑ranking – applies a more sophisticated model (cross‑encoder) to re‑order the fused list.
• LLM – receives the top‑k documents as context for answer generation.

Dense Retrieval vs Sparse Retrieval

Feature	Vector Search	Keyword Search
Understanding	Semantic similarity	Exact term occurrence
Technology	Embeddings + ANN search	BM25 / Inverted Index
Strength	Captures meaning, synonyms	Precision on codes, terms
Weakness	Rare terms, identifiers	Synonyms, paraphrasing
Best Use	General knowledge retrieval	Technical, compliance lookups

Combining them leverages the strength of each: semantic breadth from vectors, and pinpoint accuracy from keywords.

Keyword Retrieval Technologies

BM25 – a probabilistic relevance function based on term frequency and inverse document frequency. It scores how well a document matches a keyword query, with parameters to tune term saturation and document length normalization. BM25 is the default scoring in Elasticsearch and Lucene.

Inverted Index – the fundamental data structure mapping each unique word to the list of documents containing it. Search engines use this to rapidly intersect posting lists for multi‑term queries.

Common implementations accessible from Spring AI pipelines include:

• Elasticsearch – distributed search engine with rich query DSL.
• OpenSearch – open‑source fork with vector and keyword capabilities.
• Lucene – embedded library for building custom search indexes.

These can be deployed alongside a vector database or used within a single store that supports hybrid search natively (e.g., OpenSearch 2.9+, Elasticsearch 8.0+, or PGVector combined with tsvector).

Vector Retrieval Technologies

Vector databases optimized for ANN search include:

• PGVector – PostgreSQL extension supporting HNSW and IVFFlat indexes.
• Milvus – cloud‑native, open‑source, with rich filtering and multiple index types.
• Pinecone – fully managed, serverless vector search.
• Redis Vector – in‑memory, low‑latency vector operations.
• OpenSearch Vector – k‑NN plugin for the search engine.

Spring AI’s VectorStore interface provides a uniform API over these stores, making it possible to plug in hybrid search at the storage level if the underlying database supports both modalities.

Hybrid Search with Spring AI

Spring AI does not mandate a specific hybrid search engine. Instead, it provides the integration points through which you can assemble a hybrid pipeline:

• Document – universal text container with metadata.
• EmbeddingModel – portable embedding generation.
• VectorStore – vector persistence and ANN query.
• Advisors / Retrieval components – orchestration hooks for custom retrieval logic.

In practice, many Spring AI applications implement hybrid search by:

1. Querying a VectorStore for the top‑k dense results.
2. Querying an external search service (Elasticsearch, OpenSearch) for the top‑k sparse results.
3. Merging and re‑ranking the combined list.

Implementing Hybrid Retrieval with Spring AI

Below is a conceptual example. Assume vectorStore and keywordSearchService are available beans.

// 1. Dense retrieval
SearchRequest vectorRequest = SearchRequest.builder()
        .query("Spring AI ChatClient streaming")
        .topK(20)
        .similarityThreshold(0.7)
        .build();
List<Document> vectorResults = vectorStore.similaritySearch(vectorRequest);

// 2. Sparse retrieval (custom keyword service)
List<Document> keywordResults = keywordSearchService.search(
        "ChatClient streaming", 20);

// 3. Fusion
List<Document> merged = ResultFusion.merge(vectorResults, keywordResults);

A ResultFusion implementation might normalize scores, deduplicate by document ID, and apply a weighted sum or reciprocal rank fusion. The final candidate list is then passed to a re‑ranking model before feeding into the ChatClient.

Result Fusion Strategies

Weighted Score Fusion

Assigns a weight to each retrieval score and sums them:

FinalScore = α * VectorScore + (1 - α) * KeywordScore

For example, α = 0.7 gives more weight to semantic relevance, while α = 0.3 emphasizes exact matches. Weights should be tuned based on retrieval evaluation metrics.

Reciprocal Rank Fusion (RRF)

A popular, parameter‑free method that only considers the rank positions, not absolute scores:

RRFscore = Σ 1 / (k + rank_i)

where k is a constant (often 60). RRF works well when score distributions differ between retrieval sources and can be directly applied to top‑k lists.

Hybrid Search with Metadata Filtering

Metadata filtering further refines the candidate set before retrieval. A typical enterprise query might combine:

Metadata Filter (department = engineering, year >= 2025)
         +
Hybrid Search (vector + keyword)
         +
Re-ranking

In Spring AI, metadata filters are expressed as filter expressions within the SearchRequest:

SearchRequest request = SearchRequest.builder()
        .query("Spring AI performance tuning")
        .filterExpression("department == 'engineering' && year >= 2025")
        .topK(20)
        .build();

The vector store applies these filters during the ANN search, ensuring only authorized and relevant segments are considered. The keyword search system should apply the same filters for consistency.

Enterprise Hybrid RAG Architecture

A production hybrid RAG system integrates multiple specialized components:

Key enterprise components:

• Document Processing – Loads, cleans, and chunks documents; also updates keyword indexes.
• Embedding Pipeline – Generates and updates vectors.
• Keyword Index – Maintains inverted index (via Lucene, Elasticsearch).
• Vector Database – Stores embeddings with metadata.
• Hybrid Retrieval Layer – Orchestrates parallel queries and fusion.
• Re‑ranking Service – Optional cross‑encoder for final ordering.
• Security Layer – Enforces access control at filter level.
• Monitoring – Tracks latency, recall, and fusion effectiveness.

Performance Considerations

• Latency – Execute vector and keyword queries in parallel to minimize wall‑clock time. Cache embeddings for frequent queries.
• Index Management – Keep both vector and keyword indexes synchronized. Incremental updates are essential for large‑scale deployments.
• Scaling – Distribute vector search across shards and scale keyword search horizontally via distributed search engines.
• Cost – Monitor token usage for embedding generation and re‑ranking API calls. Consider lighter embedding models for query‑side when appropriate.

Common Problems and Solutions

Problem	Cause	Solution
Too many irrelevant results	Poor fusion weights	Tune fusion strategy with evaluation
Missing exact matches	Weak keyword search	Improve keyword index, use BM25 tuning
Slow queries	Sequential retrieval	Parallelize dense and sparse retrieval
Duplicate documents	Multiple retrieval paths	Deduplicate by unique document ID
Inconsistent ranking	Different score ranges	Normalize scores or use RRF

Hybrid Search Best Practices

• Use vector search for semantic breadth and synonym handling.
• Use keyword search for exact matches on identifiers, codes, and product names.
• Apply metadata filtering early to reduce the retrieval space and improve security.
• Add re‑ranking for high‑stakes applications where result order significantly impacts answer quality.
• Monitor retrieval metrics (recall, precision, nDCG) and run A/B tests when changing fusion parameters.
• Evaluate with real user queries, not just synthetic benchmarks, to ensure practical performance.

Relationship With Other Spring AI Components

• Spring AI RAG – Hybrid search is a key retrieval strategy within the broader RAG architecture.
• Embedding Pipeline – Provides the dense vectors that power the vector search branch.
• Metadata Filtering – Works in tandem with hybrid retrieval to scope results.
• Vector Databases – The storage layer that serves as the vector search engine and may provide native hybrid capabilities.

These components are designed to be composable, so you can implement hybrid search at any level of the stack.

Interview Questions

Spring AI Hybrid Search Interview Questions

1. Why is vector search insufficient for enterprise RAG?
   Vector search can miss exact technical terms, identifiers, and codes critical for compliance and precision. Hybrid search adds keyword matching to address this.

2. What is the difference between dense and sparse retrieval?
   Dense retrieval uses embeddings to capture semantic similarity; sparse retrieval uses keyword matching (BM25, inverted index) for term‑level precision.

3. How does hybrid search improve RAG accuracy?
   It combines semantic relevance with exact matching, ensuring that retrieved documents are both topically appropriate and contain the precise terms mentioned in the query.

4. What is Reciprocal Rank Fusion (RRF)?
   RRF merges ranked lists by summing the reciprocal of the rank position, ignoring absolute scores. It is simple, robust, and effective when combining heterogeneous retrieval sources.

5. How would you design a production hybrid retrieval system?
   I would use parallel retrieval from a vector store and a keyword search engine, apply metadata filters consistently, fuse results with RRF or weighted scoring, re‑rank with a cross‑encoder, and instrument the pipeline with monitoring for recall and latency. I’d also plan incremental index updates and caching.