Skip to main content

Spring AI Chunking Strategies for RAG

Chunking—the process of splitting a long document into smaller, semantically manageable pieces—is one of the most impactful design decisions in a retrieval‑augmented generation (RAG) system. The size, shape, and overlap of these chunks directly influence retrieval precision, response quality, and cost. Spring AI provides a flexible set of abstractions to implement chunking strategies that match your document types and retrieval goals.

This guide explores why chunking matters, how it works inside a Spring AI pipeline, and which strategies to apply for different enterprise scenarios. You will learn how to configure and evaluate chunking, avoid common pitfalls, and build a chunking architecture that scales.

Why Does RAG Need Chunking?​

Large language models and embedding models have strict input limits. Sending an entire 100‑page PDF in one prompt is impossible—and even if it were, retrieval would suffer. Chunking solves several problems:

  • Context window limitations: Embedding models (e.g., text-embedding-ada-002) typically accept 8,191 tokens or fewer. A single long document must be split to fit.
  • Token costs: Smaller chunks reduce the number of tokens sent to the embedding model and the LLM during generation.
  • Retrieval precision: A search for a specific paragraph should return that paragraph, not a 20‑page section containing irrelevant material.
  • Response quality: The LLM can better extract a precise answer from a few tightly focused chunks than from a wall of text.

In short, better chunks produce better retrieval. The transformation from raw document to high‑quality vector index depends on how you slice the text.

How Chunking Works in a RAG Pipeline​

In a Spring AI application, chunking sits between document cleaning and embedding generation:

Spring AI abstracts the splitting logic through the DocumentTransformer interface. The TokenTextSplitter is the primary implementation, but you can write custom transformers to implement any strategy. Once split, each chunk becomes a separate Document object with its own text and inherited metadata. These documents are then embedded and stored in a VectorStore.

Chunk Size Explained​

Chunk size is usually measured in tokens because that’s the unit embedding models count. It represents the approximate number of tokens in each chunk after splitting. There is a trade‑off:

Small Chunks​

  • Advantages: More precise retrieval—the returned chunk is exactly what the user needed. Less irrelevant context reduces hallucination risk.
  • Disadvantages: Loss of surrounding context; a key detail might be split across two chunks. More chunks mean higher storage and indexing costs.

Large Chunks​

  • Advantages: Preserve context; the model can see a full thought or section. Fewer chunks reduce vector database size.
  • Disadvantages: Lower retrieval precision—irrelevant content may be bundled. Higher token consumption in prompts.

A common starting point is 500–1,000 tokens per chunk, but the optimal size depends on the document type and embedding model.

Chunk Overlap Explained​

Overlap means that adjacent chunks share a small amount of text at their boundaries. This prevents the loss of context that might be split mid‑sentence.

Without overlap:

Chunk 1: Spring AI provides AI abstractions for Java.
Chunk 2: ChatClient simplifies application development.

With overlap (2‑token):

Chunk 1: Spring AI provides AI abstractions for Java.
Chunk 2: abstractions for Java. ChatClient simplifies application development.

Too little overlap risks cutting a thought in half; too much overlap wastes token budget and can cause duplicate retrieval. Overlap of 10–20% of the chunk size is typical.

Common Chunking Strategies​

Different strategies suit different document types. Spring AI’s TextSplitter variants support several of these.

Fixed‑Size Chunking​

Splits text after a fixed number of characters or tokens, without respect for structure.

  • Advantages: Simple, fast, predictable.
  • Disadvantages: Cuts sentences and paragraphs arbitrarily; poor semantic quality.
  • Use cases: Large, homogeneous corpora where structure is uniform, or as a baseline.

Spring AI’s TokenTextSplitter can be configured with a fixed token length and overlap.

Sentence‑Based Chunking​

Splits at sentence boundaries, ensuring that each chunk ends at a natural linguistic break.

  • Advantages: Better readability and semantic coherence.
  • Disadvantages: Variable chunk sizes; some sentences may be too long or too short.
  • Use cases: Legal documents, reports, and any text where sentence meaning must stay intact.

You can implement sentence‑aware chunking by extending TokenTextSplitter to use sentence separators.

Paragraph‑Based Chunking​

Chunks correspond to paragraphs—a natural unit of thought in many document formats.

  • Advantages: High semantic integrity; aligns with how documents are written.
  • Disadvantages: Inconsistent chunk sizes; some paragraphs may be very long.
  • Use cases: Articles, knowledge bases, technical documentation.

When using Markdown or HTML, structure‑aware chunking can preserve paragraphs and headings.

Recursive Chunking​

A hierarchical approach: apply a series of separators (e.g., double newline, single newline, period) recursively until the chunk fits the size limit.

Document
  → Sections (split by headings)
    → Paragraphs (split by blank lines)
      → Sentences (split by periods)
        → Words (final fallback)

This is the most robust general‑purpose strategy and is the basis for TokenTextSplitter’s recursive mode in some implementations. It works well for enterprise documents with mixed structure.

Semantic Chunking​

Uses embedding similarity to group sentences into chunks that share a common topic.

  • Advantages: Very high retrieval relevance; chunks are inherently semantic.
  • Disadvantages: Computationally expensive during ingestion; less predictable sizes.
  • Use cases: Complex, unstructured knowledge where topic boundaries are fuzzy.

Spring AI does not include a built‑in semantic chunker, but you can build one by using an EmbeddingModel inside a custom DocumentTransformer.

Structure‑Aware Chunking​

Preserves the logical structure of a document—headings, tables, code blocks—so that each chunk retains its section context.

  • Markdown: Split by headings (#, ##); keep the heading as prefix.
  • HTML: Split by section tags; remove navigation noise.
  • PDF: Use page boundaries and layout detection.

For technical documentation and API references, structure‑aware chunking dramatically improves retrieval because a chunk like “Authentication → OAuth2 details” carries its own context.

Code Chunking for Developer Knowledge Bases​

When indexing source code, API docs, or technical manuals, special rules apply:

  • Split by class or method boundaries in code.
  • Keep code and its associated comments together.
  • Preserve the fully qualified name as metadata.

A chunk of Java code should ideally contain one logical unit (a method or a small class), with the import context prepended if needed.

Spring AI TextSplitter​

Spring AI provides the TextSplitter abstraction (via DocumentTransformer) and a primary implementation, TokenTextSplitter.

Key configuration parameters:

  • defaultChunkSize – target number of tokens per chunk.
  • minChunkSizeChars / minChunkLengthToEmbed – avoid embedding extremely small fragments.
  • keepSeparator – whether to retain the split character.
  • chunkOverlap – number of tokens to overlap between chunks.
  • separators – a list of strings to try in order (recursive splitting).

A typical Spring AI bean definition:

@Bean
DocumentTransformer documentSplitter() {
    return new TokenTextSplitter(
        800,  // defaultChunkSize
        80,   // chunkOverlap
        10,   // minChunkSizeChars
        50,   // minChunkLengthToEmbed
        List.of("\n\n", "\n", ". ", " ")  // separators
    );
}

This transformer can be applied to a list of Document objects:

List<Document> splitDocs = documentSplitter.apply(rawDocuments);

The split documents retain the original metadata and can be enriched with chunk index and parent document ID.

Spring AI Chunking Example​

A complete ingestion snippet might look like this:

// 1. Load raw documents
DocumentReader reader = new TikaDocumentReader(
        new FileSystemResource("docs/"));
List<Document> raw = reader.read();

// 2. Clean and split
DocumentTransformer cleaner = new CleanupTransformer(); // custom
DocumentTransformer splitter = new TokenTextSplitter();
List<Document> chunks = splitter.apply(cleaner.apply(raw));

// 3. Embed and store
EmbeddingModel embeddingModel = ...; // auto-configured
VectorStore vectorStore = ...;       // auto-configured
vectorStore.add(chunks);

The chunks are now indexed and ready for retrieval. The VectorStore.add() method handles embedding generation internally for each chunk if the documents already contain embeddings; otherwise, you must embed them first.

Choosing Chunk Size​

There is no universal ideal chunk size, but the following table offers starting points:

ScenarioRecommended Strategy
Technical documentation (Markdown, headings)Structure‑aware + 500–800 tokens, overlap 100
Legal documentsSentence‑based, 300–500 tokens, overlap 50
Product manualsRecursive chunking, 600–1,000 tokens, overlap 150
Chat historyUtterance‑based (each message a chunk)
Source codeMethod‑ or class‑based, no fixed token limit
General knowledge baseRecursive chunking, 800 tokens, overlap 80

Factors to consider: model context window, embedding model, document type, and the expected queries.

Chunking Evaluation​

You can objectively measure the impact of chunking choices:

  • Retrieval accuracy: Use a labeled dataset to compare which chunks are returned vs. the ideal.
  • Answer correctness: With a fixed LLM, measure how often the answer is factually correct.
  • Context relevance: Manually score whether the retrieved chunks contain the answer.
  • Latency and token usage: Monitor embedding time and prompt token count.

Offline evaluation with a gold‑standard Q&A set is the best way to tune parameters before production.

Advanced Chunking Techniques​

Parent‑Child Retrieval​

Small chunks are used for retrieval (high precision), but the parent (larger) chunk is sent to the LLM for generation (preserving context). Spring AI supports this pattern by storing both levels and using metadata to link them.

Sliding Window Chunking​

A fixed‑size window slides over the text with a step smaller than the window, creating highly overlapping chunks. This is useful for very dense text where every word matters.

Query‑Aware Chunking​

Chunks are created based on the anticipated questions. For example, an FAQ document might be chunked by individual Q&A pairs.

Hybrid Chunking​

Combine multiple strategies: use structure‑aware splitting for sections, then recursive splitting within each section, finally applying semantic merging. This yields the most robust results for diverse enterprise corpora.

Common Chunking Mistakes​

  • Chunk size too large: Retrieval becomes coarse; irrelevant text drowns the answer.
  • Chunk size too small: Answers are fragmented; critical context is lost across chunks.
  • Ignoring metadata: Without document source and section tags, retrieved chunks are opaque and hard to filter.
  • Splitting without understanding the document structure: Breaking a table or code block in half destroys its meaning.
  • Applying one strategy blindly: A PDF of contracts behaves differently from a Markdown wiki.

Enterprise Chunking Architecture​

In a production system, chunking is part of a larger ingestion pipeline:

The chunking engine may dynamically choose a strategy based on file type or metadata. The pipeline should be asynchronous, fault‑tolerant, and horizontally scalable. For large‑scale ingestion, use a message queue and parallel workers.

Best Practices​

  • Start with document‑aware chunking: use headings, sections, and paragraphs.
  • Measure retrieval quality with a representative query set and adjust parameters iteratively.
  • Keep metadata with chunks: source, section title, page, chunk index.
  • Preserve document hierarchy: use parent‑child retrieval if needed.
  • Tune chunk size experimentally: run A/B tests comparing different configurations.
  • Avoid one‑size‑fits‑all: implement a strategy selector based on content type.
  • Monitor token usage and chunk count to control costs.

What's Next​

Chunking is a core component of the RAG section, and its proper design sets the stage for retrieval success.

Key Takeaways​

  • Chunking directly determines the granularity of vector search and the quality of RAG responses.
  • Spring AI’s TokenTextSplitter offers configurable, recursive, and token‑aware splitting.
  • Different document types demand different strategies: structure‑aware for docs, sentence‑based for legal, etc.
  • Overlap prevents information loss at boundaries.
  • Evaluate chunking choices with real queries; do not guess.
  • In production, build a flexible, observable ingestion pipeline that can adapt to new document types.