Spring AI Framework

The Spring AI Framework section is the architectural heart of this handbook. It documents the core abstractions, execution pipelines, and programming models that every Spring AI application depends on. While the Getting Started section gives you a running application and the Provider section tells you how to connect to specific LLM services, this section explains what happens between your code and the model—and why it is designed that way.

Spring AI is not a thin HTTP client. It is a layered framework that applies familiar Spring patterns—templates, interceptors, converters, and dependency injection—to the unique challenges of AI integration. Understanding this layer is essential if you want to write testable, portable, and production-ready AI features. It is also the prerequisite for advanced topics like RAG pipelines, enterprise AI architecture, and source code analysis.

This landing page serves as your navigation hub and conceptual map. It introduces the major framework components, explains how they fit together, and recommends a learning order that builds knowledge incrementally.

Why the Spring AI Framework Matters

Enterprise Java teams do not succeed by calling raw REST APIs from business logic. They succeed by building on abstractions that provide consistency, composability, and operational visibility. The Spring AI framework layer exists to provide exactly that for AI workloads.

• Spring-style abstraction and consistency – Whether you are working with OpenAI, Azure OpenAI, or a local Ollama model, you program against the same ChatModel interface. This consistency reduces cognitive load and makes it easy to onboard new team members.
• Framework-level composability – Advisors, memory, and tool calling are not afterthoughts. They are first-class framework components that can be combined declaratively. You can add logging, chat history, and function execution to any ChatClient request without rewriting your application code.
• Vendor independence – The framework decouples your business logic from provider-specific APIs. Changing a model provider often requires only configuration changes, not code rewrites. This gives you the freedom to choose the best model for each task and the flexibility to adapt as the AI landscape evolves.
• Production-oriented design – Features like retry, token usage tracking, structured output, and observability are built into the framework, not bolted on. This means production concerns are addressed from the first line of code, not retrofitted under pressure.
• Reduced integration complexity – The framework handles prompt assembly, response parsing, streaming, and error recovery. What would otherwise be hundreds of lines of bespoke integration code becomes a few method calls on well-defined interfaces.

By the time you finish this section, you will see Spring AI not as a library for calling LLMs, but as a complete, opinionated framework for building intelligent systems—one that aligns perfectly with the Spring philosophy.

Framework Mental Model

Think of Spring AI as a layered framework where each layer has a distinct responsibility. Understanding this mental model will help you navigate the component documentation and reason about application design.

Application Layer
Your Spring Boot services, controllers, and business logic. This layer interacts with AI capabilities primarily through ChatClient. It does not know about provider specifics.

ChatClient
The primary entry point for conversational AI. It orchestrates the assembly of prompts, the application of advisors, the invocation of tools, and the handling of responses. In most applications, ChatClient is the only API you need to use directly.

Prompt / PromptTemplate / Advisors / Memory / Tool Calling
These are the mechanisms that enrich a simple user message with structure and behavior:

• Prompt and PromptTemplate define what is sent to the model, including system instructions, user messages, and format requirements.
• Advisors intercept and modify requests and responses. They are the foundation for cross-cutting concerns like logging, content filtering, and retrieval augmentation.
• Memory maintains conversation state across multiple interactions, enabling stateful chatbots and agents.
• Tool Calling allows the model to invoke Java methods to retrieve data or perform actions, extending the model’s capabilities beyond its training data.

ChatModel / EmbeddingModel
The portable interfaces that abstract the actual AI model. ChatModel represents a language model that can generate completions; EmbeddingModel represents a model that can produce vector embeddings. Every provider implements these interfaces.

Provider Integrations
Concrete implementations for OpenAI, Azure OpenAI, Ollama, Anthropic, and others. They translate Spring AI requests into provider-specific API calls and responses back into Spring AI types.

This layering means you can test each component independently, swap providers at the model layer, and add cross-cutting behavior via advisors—all without modifying the layers above.

Core Framework Components

The table below lists the primary framework components covered in this section. Each component has a dedicated guide that explains its API, configuration, and design rationale.

Component	Responsibility	Related Guide
ChatClient	Primary entry point for building and executing AI requests	chat-client
Prompt	Representation of the complete input to a model, including system and user messages	prompt
PromptTemplate	Templating mechanism for creating prompts with dynamic variables	prompt-template
ChatModel	Portable abstraction for invoking a chat-based language model	chat-model
ChatResponse	Container for the model’s response, including message content and metadata	chat-response
EmbeddingModel	Portable abstraction for generating vector embeddings from text	embedding-model
Advisors	Interceptor chain for modifying requests and responses (e.g., logging, RAG, content filtering)	advisors
Memory	Abstraction for storing and retrieving conversation history	memory
Streaming	Support for receiving model responses as a stream of tokens	streaming
Structured Output	Mechanisms for converting raw model responses into typed Java objects	structured-output
Tool Calling	Framework for allowing models to invoke pre-registered Java methods	tool-calling
Evaluation	Utilities for assessing the quality and correctness of AI responses	evaluation
Observability	Metrics, tracing, and monitoring for AI interactions	observability
Retry	Strategies for retrying failed AI requests with backoff and circuit breaking	retry
Token Usage	Tracking and managing token consumption across requests	token-usage

Each guide can be read independently, but the recommended order below ensures you build a coherent understanding.

Recommended Learning Order

The framework components are interconnected. Learning them in the right sequence prevents confusion and allows each concept to build on the previous one.

1. ChatClient – Start here. ChatClient is the facade through which most framework features are accessed. Understand its builder API and how it wires together other components.
2. Prompt – Learn the structure of a request: system messages, user messages, and the role they play in guiding model behavior.
3. PromptTemplate – Add dynamism to your prompts by using templates with placeholders. This is the foundation for reusable prompt engineering.
4. ChatModel – Move one layer down to understand the portable model interface and how different providers implement it.
5. ChatResponse – Understand what comes back from the model: message content, metadata, token usage, and finish reasons.
6. EmbeddingModel – Extend your knowledge to embedding models, which are essential for semantic search and RAG.
7. Advisors – Learn the interceptor pattern that powers logging, content filtering, and retrieval augmentation.
8. Memory – Add conversation history to your application, enabling multi-turn interactions.
9. Streaming – Handle token-by-token responses for real-time user experiences.
10. Structured Output – Convert model text into typed Java objects, bridging the gap between AI generation and enterprise data models.
11. Tool Calling – Enable models to invoke your Java methods, unlocking dynamic data retrieval and action execution.
12. Evaluation – Measure the quality of your AI responses programmatically.
13. Observability – Instrument your AI interactions for production monitoring and debugging.
14. Retry – Build resilience by configuring retry and fallback strategies for transient failures.
15. Token Usage – Monitor and optimize the cost and performance of your AI requests.

This order is intentional: it starts with the developer-facing entry point and gradually moves into cross-cutting concerns and operational topics.

Framework Architecture Overview

From an architecture perspective, the Spring AI framework implements a classic pipeline pattern with well-defined extension points.

Request flow:

1. Application code creates a request via ChatClient.Builder, specifying prompt, tools, advisors, and memory.
2. The advisor chain processes the request in order, allowing each advisor to modify the prompt or enrich the context.
3. The final prompt is passed to a ChatModel implementation, which converts it into a provider-specific API call.
4. The provider returns a raw response, which is packaged as a ChatResponse by the model adapter.
5. Post-processing components—structured output converters, tool call executors—act on the response.
6. The final result is returned to the application as a typed ResponseEntity or as a raw ChatResponse.

Key design characteristics:

• Abstractions at stable boundaries – ChatModel and EmbeddingModel are stable interfaces; provider implementations are isolated behind them.
• Providers hidden behind interfaces – Application code never interacts with provider APIs directly. This enables testing with mock models and seamless provider swaps.
• Extension and testing support – Advisors, converters, and tool callbacks are all replaceable. The framework is designed for unit testing each component in isolation.

This architecture is what makes Spring AI more than a convenience wrapper: it is a structured foundation for building robust, evolvable AI systems.

Core Concepts Readers Must Understand

Each of the following concepts is a pillar of the framework. You will encounter them throughout the handbook, and a solid grasp of each one is necessary for effective application design.

ChatClient

ChatClient is the primary API for interacting with a chat model. It provides a fluent builder for constructing requests that include prompts, tools, advisors, and memory. Under the hood, it coordinates the entire execution pipeline. In most applications, you should inject and use ChatClient rather than calling ChatModel directly.

Prompt

A Prompt is the complete input sent to a language model. It consists of one or more Message objects, which can be of type SystemMessage, UserMessage, or AssistantMessage. The prompt defines the conversation context and the instruction the model should follow. Mastering prompt structure is the first step toward controlling model behavior.

PromptTemplate

PromptTemplate allows you to define a prompt as a template with placeholder variables. This is the Spring AI equivalent of a prepared statement. Templates can be stored in resource files, externalized from code, and populated dynamically at runtime. They are essential for maintaining clean, testable prompt engineering workflows.

ChatModel

ChatModel is the portability interface that abstracts a language model. Its primary method, call(Prompt), returns a ChatResponse. Provider implementations (OpenAI, Ollama, etc.) implement this interface. By programming against ChatModel, your application remains decoupled from any specific AI service.

ChatResponse

ChatResponse encapsulates the model’s output. It contains one or more Generation objects, each with a Message and metadata. The response includes the completion text, the reason the model stopped (e.g., STOP, LENGTH, TOOL_CALL), and token usage statistics. Understanding this structure is key to post-processing and logging.

EmbeddingModel

EmbeddingModel is the abstraction for models that convert text into numerical vectors. It exposes methods like embed(String text) and embed(Document document). Embeddings are the foundation for semantic search, clustering, and RAG retrieval.

Advisors

Advisors implement a simple interceptor pattern. An advisor can modify the Prompt before it reaches the model and can modify the ChatResponse before it reaches the caller. Common advisors include logging, chat memory injection, retrieval augmentation (RAG), and content filtering. They enable clean separation of cross-cutting concerns.

Memory

The Memory interface defines how conversation history is stored and retrieved. Implementations range from in-memory maps for simple applications to database-backed stores for long-running sessions. Memory integrates with the advisor chain to automatically include past messages in the prompt.

Streaming

Streaming support allows the model to return tokens incrementally rather than in a single response. Spring AI provides a Flux<String> or Flux<ChatResponse> that delivers tokens as they are generated. This is essential for building responsive user interfaces and for handling long-form generations.

Structured Output

Structured output converts raw model text into typed Java objects. Using BeanOutputConverter or custom converters, you can instruct the model to return JSON that matches a specific schema and have the framework deserialize it into a record or class. This bridges AI generation with enterprise data pipelines.

Tool Calling

Tool calling allows the model to decide, during generation, that it needs to invoke a registered Java method. The method’s parameters and description are communicated to the model via the @Tool annotation. The framework handles the execution of the function and feeds the result back into the model for continued generation. This is the foundation of agent-like behavior.

Framework vs Provider Integration

A common point of confusion is the boundary between the framework and provider-specific code. This distinction is fundamental to Spring AI’s design:

• Framework abstractions (this section) define what you can do: send prompts, receive responses, use memory, call tools, parse structured output. They are provider-agnostic and stable.
• Provider implementations (covered in the Providers section) define how those abstractions connect to a specific AI service: authentication, HTTP clients, request serialization, and response deserialization.

This section focuses exclusively on the abstractions and their behavior. You do not need to know anything about OpenAI’s API to understand ChatClient. You will need that knowledge when you configure a provider, but the code you write against ChatClient remains the same regardless.

Keeping this separation clear in your mind will make you a more effective Spring AI engineer. It will also make your code more testable, because you can mock ChatModel instead of a provider’s HTTP client.

Where the Framework Leads Next

The framework section is the gateway to every advanced topic in the handbook. Once you have mastered the core abstractions, you are ready to explore:

• RAG – Combines ChatClient, Advisors, EmbeddingModel, and VectorStore to build retrieval-augmented generation systems.
• Vector Databases – Configures and uses vector stores that back the VectorStore abstraction introduced here.
• Providers – Deep dives into each supported AI provider, including authentication, model options, and provider-specific features.
• Enterprise AI – Applies framework components in production: security, multi-tenancy, testing, deployment, and performance tuning.
• Tutorials – End-to-end walkthroughs that combine multiple framework components into complete applications.
• Source Code – Internal implementation details for those who need to extend or debug the framework.

Each of these sections references the framework concepts you learn here. A solid foundation makes those sections accessible and meaningful.

Who Should Read This Section?

Java Developers

If you are integrating AI into a Java application, this section gives you the vocabulary and mental model to do so correctly. It shows you how to use Spring AI as a framework rather than a thin utility, which results in cleaner, more maintainable code.

Spring Developers

You already understand the power of Spring’s abstraction layers. This section maps AI concepts onto patterns you know: templates (PromptTemplate), interceptors (Advisors), and annotated methods (@Tool). It demonstrates that Spring AI is a natural extension of the Spring ecosystem.

AI Engineers

If you have experience with Python AI frameworks, this section helps you translate that knowledge into a statically typed, enterprise-grade Java framework. You will see how concepts like chains, memory, and tool use are implemented in a Spring-native way.

Software Architects

Architects need to understand the framework’s structure to make informed design decisions. This section provides the architectural diagrams, component relationships, and extension points that support technology evaluation and system design.

Common Mistakes to Avoid

• Treating ChatModel as the main entry point – While ChatModel is the lowest-level abstraction, application code should typically use ChatClient. ChatClient adds advisor processing, memory integration, and structured output support that ChatModel alone cannot provide.
• Skipping Prompt and PromptTemplate – Hardcoding prompt strings in business logic leads to unmaintainable code. Using templates externalizes prompt engineering and makes it testable.
• Confusing framework abstractions with provider APIs – ChatModel is not a provider client. It is a portable interface. Writing code that depends on provider-specific features defeats the purpose of the abstraction layer.
• Ignoring Advisors and Memory – Cross-cutting concerns like logging, security filtering, and conversation history should not be scattered across service methods. Use advisors to centralize this logic.
• Jumping directly to RAG before understanding core abstractions – RAG is a composition of ChatClient, Advisors, EmbeddingModel, and VectorStore. Without understanding each piece, debugging RAG issues becomes extremely difficult.

Spring AI Framework in the Handbook

The Framework section sits at the center of the handbook. Every other section either prepares you for it or builds on it.

Section	Relationship to Framework
Getting Started	Provides the setup and first-application experience; introduces the need for framework abstractions.
RAG	Applies framework components (ChatClient, Advisors, EmbeddingModel) to retrieval-augmented generation.
Providers	Details the implementations behind ChatModel and EmbeddingModel.
Enterprise AI	Shows how framework components are hardened for production.
Tutorials	Hands-on recipes that combine multiple framework concepts.
Comparison	Compares the framework’s design to alternatives like LangChain4j.
Source Code	Reads the framework’s internals; requires deep understanding of the concepts described here.

Use this table to orient yourself as you move through the handbook.

Summary

The Spring AI Framework section is the core of your Spring AI knowledge. It introduces the abstractions, execution models, and design patterns that make Spring AI a framework rather than a convenience library. By mastering ChatClient, Prompt, ChatModel, Advisors, Memory, Tool Calling, and their companions, you gain the ability to design AI features that are portable, testable, and production-ready.

Take the time to understand each component and how they relate. Work through the recommended learning order. Experiment with the code examples in each guide. The investment you make here will pay dividends when you move into RAG, enterprise architecture, and source code analysis.

When you are ready, continue to the first guide: ChatClient.