Skip to main content

Spring AI Architecture: Core Components Explained

1. Introduction to Spring AI Architecture​

Spring AI is an application framework for integrating generative AI capabilities into Java and Spring applications. Its architectural purpose is not to train or serve models — it is to make enterprise Java systems use models reliably, without coupling business code to any single provider.

The problems Spring AI addresses are concrete and recurring. An application needs to talk to OpenAI today and Anthropic or a local Ollama instance tomorrow. It needs to combine model calls with enterprise data, internal APIs, and documents. It needs prompt assembly, conversation history, tool execution, retrieval, and observability to be reusable across services rather than reimplemented in each one. And it needs all of this to fit the conventions of the Spring ecosystem — dependency injection, auto-configuration, externalized configuration, and the testing model that Spring developers already use.

The core architectural idea is straightforward: provide portable abstractions and Spring-native integrations so that application code depends on interfaces, while provider SDKs, vector databases, and model backends remain replaceable implementation details. This mirrors the design philosophy of JDBC and Spring Data — standardize the common 80% (chat, embedding, vector search), and provide escape hatches for the remaining provider-specific capabilities rather than pretending they do not exist.

Spring AI 2.0, generally available since June 2026, is the reference version for this guide. It is built on Spring Boot 4.1 and Spring Framework 7, and it changed the internal architecture of tool execution in a way that matters for anyone designing agentic systems — a change covered in Section 8.

The relationship between the major layers can be stated briefly before being expanded: a Spring Boot application calls application-facing APIs (ChatClient, Advisors); those APIs depend on core model abstractions (ChatModel, EmbeddingModel, VectorStore); the abstractions are backed by provider and infrastructure integrations; and all of it sits inside a Spring Boot configuration and observability layer.

2. High-Level Spring AI Architecture​

Spring AI is best understood as a layered architecture. The layers are conceptual — they do not correspond one-to-one with Maven modules — but they clarify which component owns which responsibility.

2.1 Application and Business Layer​

This is where business capabilities live: AI-powered customer support, enterprise knowledge assistants, document analysis, intelligent search, content generation, and agent-based task automation. The architectural rule is that this layer should contain business logic and orchestration of domain services — not provider credentials, not raw HTTP calls to model APIs, and not provider-specific request objects. When business logic is coupled to a provider SDK, every model migration becomes a cross-cutting code change.

2.2 Application API Layer​

Spring AI exposes several application-facing APIs, and they are not equivalent abstractions:

  • ChatClient — the primary fluent entry point for constructing prompts and invoking chat models.
  • Advisors — middleware that intercepts and enhances interactions in the request pipeline.
  • Tool Calling — the mechanism by which models request application-defined functions.
  • RAG orchestration — document ingestion and retrieval augmentation, largely assembled from VectorStore, EmbeddingModel, and advisor-based retrieval.
  • MCP integration — clients and servers implementing the Model Context Protocol.

ChatClient is the central application API; the others are either extensions to its pipeline (Advisors, tool calling) or supporting subsystems invoked from it. Treating all of these as interchangeable "APIs" obscures the fact that Advisors and tool calling operate inside the ChatClient request path, while RAG ingestion and MCP connectivity operate alongside it.

2.3 Core Model Abstraction Layer​

Below the application APIs sit the model abstractions: ChatModel, EmbeddingModel, ImageModel, and audio-related interfaces, together with message types and prompt representations. These interfaces define synchronous and streaming invocation, model-specific options, and provider portability. Different modalities have different interfaces and execution behavior — an embedding call is not a chat call with a different return type, and the abstraction layer does not pretend otherwise.

2.4 AI Integration and Infrastructure Layer​

This layer contains the implementations that connect abstractions to reality: provider-specific chat and embedding models, vector database integrations, document readers, persistent chat memory repositories, tool callback resolvers, and MCP client/server transports. Provider implementations are where authentication, wire formats, and provider-specific quirks are confined.

2.5 Spring Boot Integration Layer​

Spring AI integrates through auto-configuration and starters. As of the 1.x line, auto-configuration moved from a single monolithic artifact to per-model and per-vector-store auto-configuration artifacts, so that an application using one provider does not pull in unrelated dependencies or inherit their version conflicts. Starters are thin dependency descriptors; auto-configuration modules create the beans; application code injects the resulting abstractions.

3. Core Architectural Building Blocks​

ComponentPrimary ResponsibilityArchitectural Role
ChatClientFluent prompt construction and request orchestrationApplication-facing interaction API
ChatModelProvider-level chat invocationModel abstraction boundary
EmbeddingModelConverting text to vector representationsSemantic encoding service
Prompt / MessageStructured request representationData model for model input
ChatResponseResponse envelope with generations and metadataData model for model output
AdvisorIntercepting and enhancing interactionsExtensibility middleware
VectorStoreStoring and similarity-searching embeddingsRetrieval infrastructure abstraction
DocumentUnit of ingested content with metadataData model for knowledge sources
ChatMemoryConversation history managementStateful context subsystem
Tool CallingModel-requested application function executionAgentic capability mechanism
MCPStandardized external tool/resource connectivityIntegration boundary

Each component occupies a distinct position. ChatClient and Advisor are interaction APIs. ChatModel, EmbeddingModel, and VectorStore are abstractions with multiple implementations. Prompt, Message, ChatResponse, and Document are data models that flow between layers. Tool calling and MCP are mechanisms rather than types. Memory is a stateful subsystem whose lifecycle is separate from any single request. Conflating these categories is the most common source of architectural misunderstanding in Spring AI applications.

4. Model API and Provider Abstraction​

The Model API is the seam between application code and AI providers. ChatModel and EmbeddingModel define portable contracts; provider modules implement them. An application service that depends on ChatModel does not know whether the request will be served by OpenAI, Anthropic, or a local model.

The benefits of this abstraction are reduced provider coupling, consistent application code, easier provider evaluation, and simpler migration when it becomes necessary. The limits are equally important to design around. Abstraction does not normalize model behavior, context-window sizes, tokenization, tool-calling semantics, structured-output enforcement, or provider-specific parameters. A portable call site is portable at the API level; it is not guaranteed to produce equivalent results across providers. Provider-specific options must be isolated in per-provider configuration rather than assumed to be universally accepted.

5. ChatClient and the Application Interaction Model​

ChatClient is the primary application-facing API for chat interactions. It supports synchronous (call()) and streaming (stream()) invocation, fluent prompt construction with system and user messages, template variables, structured output mapping, advisor registration, and tool registration.

The relationship is:

Application Service → ChatClient → (Advisor chain) → ChatModel → Provider

Application developers should generally use ChatClient rather than ChatModel directly. ChatClient supplies the advisor pipeline that carries memory, retrieval, logging, and tool execution; it applies default configuration consistently; and it provides response extraction methods that return text, typed objects, or full response envelopes. ChatModel remains appropriate for infrastructure work — custom advisors, provider adapters, diagnostics — or for low-level control that deliberately bypasses the application pipeline. The detailed API reference belongs to the Spring AI ChatClient guide; architecturally, the point is that ChatClient is the composition point where independently developed capabilities are assembled into a single request.

6. Prompt and Message Architecture​

A prompt in Spring AI is a collection of typed messages, not a single string. System messages carry behavioral instructions, user messages carry the request, assistant messages carry prior model turns, and tool-related messages carry tool call requests and results. Prompt pairs the message list with ChatOptions such as model name and temperature.

This structured representation is what makes conversation history, multi-turn tool execution, and template substitution coherent. It also creates a practical hazard: prompts containing literal JSON can collide with the default {} template delimiter, so applications that embed JSON payloads must escape braces or configure a different template renderer.

7. Advisors and the Extensibility Model​

The Advisors API is Spring AI's extension mechanism. Advisors intercept the interaction pipeline, can inspect and modify the outgoing prompt, can block or short-circuit the chain, and can process the response on the way back. The chain is ordered by each advisor's getOrder() value — lower values execute earlier on the request path and later on the response path, so the first advisor to see a request is the last to see its response.

Advisors are how recurring capabilities become composable rather than duplicated:

  • Chat memory — loads and persists conversation history.
  • RAG — retrieves context and augments the prompt before model invocation.
  • Logging and inspection — records or audits interactions.
  • Context enrichment — injects tenant, user, or policy context.
  • Custom processing — implements planning, reflection, or validation loops.

Ordering is a deliberate design decision, not a registration detail. A memory advisor that must see the final assembled prompt should not be positioned inside an advisor that rewrites the prompt afterward. The built-in advisor set and exact execution semantics should be verified against the current release; the architectural principle — middleware composition with explicit precedence — is stable.

8. Tool Calling Architecture​

Tool calling is the mechanism by which a model requests execution of an application-defined function. The division of responsibility is unambiguous: the model decides whether a tool is needed and supplies arguments; the application executes the tool and returns the result. The model never receives direct access to the APIs behind the tools.

Spring AI 2.0 introduced a structural change in how this loop is executed. In Spring AI 1.x, tool execution loops lived inside each ChatModel implementation. In 2.0, the loop became a first-class component of the ChatClient advisor chain: ChatClient auto-registers a ToolCallingAdvisor, which drives iterations until the model produces a response without tool calls, delegating actual execution to a ToolCallingManager. ToolCallingAdvisor implements both CallAdvisor and StreamAdvisor, and carries a ToolAdvisor marker interface that DefaultChatClient uses to enforce a single tool advisor in the chain.

This matters architecturally because tool execution is now composable with other advisors through ordering. A memory advisor placed outside the tool loop sees one logical turn; an advisor placed inside the loop sees each tool round-trip. The default order places ToolCallingAdvisor at HIGHEST_PRECEDENCE + 300, above the default memory advisor at HIGHEST_PRECEDENCE + 200, which puts memory outside the tool loop by default.

The security boundary belongs to the application, not the framework. Tool arguments are model-generated and must be treated as untrusted input; authorization must be enforced independently of the model's request; and only narrowly scoped tools appropriate to a request should be registered. The Tool Calling Source Code Analysis covers the internal loop mechanics.

9. RAG Architecture in Spring AI​

Retrieval-augmented generation in Spring AI is two distinct pipelines that share an embedding model and a vector store.

9.1 Document Ingestion Pipeline​

The ETL pipeline moves raw content into a vector store: a DocumentReader (implementing Supplier<List<Document>>) loads content, a DocumentTransformer or text splitter chunks it, an EmbeddingModel converts chunks to vectors, and a VectorStore (implementing DocumentWriter) persists them.

9.2 Query Pipeline​

At request time, the user query is embedded, a similarity search with optional metadata filtering retrieves relevant chunks, the context is assembled into the prompt, and the model generates a response. In advisor-based RAG, this entire retrieval-and-augmentation stage is encapsulated in an advisor, so the application service still calls ChatClient in the ordinary way.

9.3 Architectural Variations​

Basic retrieval, advisor-based integration, modular pipelines, hybrid retrieval, re-ranking, and metadata filtering are all supported patterns. The architectural constraint is that retrieval and generation are separate stages with separate failure modes: poor retrieval produces confident but unsupported answers, and no amount of generation-side tuning fixes it. The Spring AI RAG guide covers the implementation patterns in depth.

10. Embedding Model and Vector Store Architecture​

EmbeddingModel and VectorStore are deliberately separate abstractions with separate responsibilities. EmbeddingModel converts text to vectors; VectorStore stores vectors and performs similarity search. VectorStore does not generate embeddings — it depends on an EmbeddingModel for that.

This separation has architectural consequences. The embedding model used at ingestion time must match the one used at query time, or similarity scores become meaningless. Vector dimensions, distance metrics, and normalization must be consistent between write and read paths. The choice of vector database is an infrastructure decision that should not leak into application code, which is why the VectorStore interface includes a portable, SQL-like metadata filter abstraction.

Architectural considerations include embedding model compatibility, index configuration, metadata schema design, data freshness and re-ingestion strategy, and persistence and backup of the vector index. The VectorStore API guide and Vector Databases sections cover provider selection.

11. Chat Memory and Conversation Architecture​

A chat model invocation is stateless unless context is explicitly supplied. Spring AI separates conversation memory from workflow state — memory is about what was said, not about where a process is. ChatMemory provides the logical layer for conversation history and window management, while a ChatMemoryRepository provides the storage layer; the two are decoupled so that memory policy and persistence can evolve independently. Integration with ChatClient is through a memory advisor such as MessageChatMemoryAdvisor, which loads history before the call and persists the turn after it.

Memory requires a conversation identifier supplied at request time; without it, the advisor cannot resolve which history to load. Context-window management is a separate concern: unbounded history eventually exceeds the model's limit, so windowed memory implementations bound retained messages and apply eviction policies. In-memory repositories are suitable for development but not for distributed production systems, where a shared persistent store is required for isolation, retention, and deletion. The Memory Source Code Analysis covers implementation details.

12. MCP Architecture and External System Integration​

The Model Context Protocol standardizes how AI applications discover and consume tools, resources, and prompts from external systems. Spring AI implements both sides of the client-server architecture: an MCP client maintains a 1:1 connection to an MCP server, discovers its capabilities, and forwards calls; an MCP server exposes application capabilities to MCP clients. Transport options supported by the current release include stdio and HTTP/SSE.

MCP differs from traditional Spring AI tool calling in scope. A locally defined @Tool is an in-process function registered directly on a ChatClient request. An MCP tool is discovered from an external server over a protocol boundary — which means the security and trust model changes. Capabilities consumed from an external MCP server are not under the application's control in the same way as local tools, and authorization, input validation, and audit logging must account for the external boundary.

MCP expands the integration boundary from "functions inside this application" to "capabilities exposed by any MCP-compliant system," while leaving the execution-authority principle intact: the model requests, the infrastructure executes under application-controlled policy.

13. Spring Boot Auto-Configuration and Starter Architecture​

Spring AI's developer experience is built on Spring Boot conventions. The workflow is: add the appropriate starter, configure provider credentials and model settings in application.yml, let auto-configuration create the beans, inject the abstractions into application services, and override configuration only where the defaults are insufficient.

The architectural detail that matters is granularity. Auto-configuration is no longer a single monolithic artifact; it is split per model, per vector store, and per component. This minimizes transitive dependency conflicts — a known problem with libraries that bundle competing gRPC and protobuf versions — and keeps the application's dependency graph aligned with what it actually uses. Starters are opinionated dependency descriptors; they do not contain implementation code. They pull the auto-configuration module for their component, and the auto-configuration module decides which beans to create based on classpath presence and configuration properties.

Manual configuration remains appropriate when defaults do not fit: multiple clients for the same model type, provider-specific client customization, or explicit bean wiring in tests. The distinction between Spring AI's conceptual layers and Spring Boot's configuration mechanism is important — auto-configuration does not remove the need to design the architecture; it only removes boilerplate for the parts that follow convention.

14. Observability and Evaluation Architecture​

Spring AI provides auto-configured observability through Micrometer. ChatClient, ChatModel, EmbeddingModel, and VectorStore interactions are wrapped in Micrometer Observation instances, producing metrics and traces that integrate with the Spring Boot observability ecosystem and export to Prometheus, Datadog, Grafana, or OpenTelemetry collectors. Provider integrations add HTTP-layer spans alongside the AI-operation spans, so a single logical call can be traced from the application boundary through the provider HTTP attempt.

Advisors also participate in the observability stack, which means the advisor pipeline is inspectable as part of the request lifecycle rather than as a black box. The principal operational risk is sensitive data in prompts: logging prompt and completion content without masking exposes personal data, credentials embedded in context, or proprietary business content. Token usage and model metadata are generally safer to record than raw prompt text.

Evaluation is a separate concern from monitoring. Monitoring tells you whether the system is healthy; evaluation tells you whether the output is good. Spring AI's evaluation capabilities should be verified against the current release before designing around specific classes, as the feature scope has evolved across versions. Conflating the two leads to dashboards that show green while answer quality degrades.

15. Putting the Components Together: End-to-End Architecture​

A realistic enterprise composition — an internal knowledge assistant — exercises most of the components described above.

Knowledge ingestion path: source documents are read by DocumentReader implementations, split into chunks, embedded by an EmbeddingModel, and written to a VectorStore.

User interaction path: a REST controller receives a request and delegates to an application service. The service invokes ChatClient, which constructs a Prompt and runs the advisor chain. A memory advisor attaches conversation history; a RAG advisor retrieves relevant chunks and augments the prompt; ToolCallingAdvisor handles any model-requested function calls (including MCP-discovered tools); ChatModel invokes the provider. The response returns through the advisor chain, and observability instrumentation records metrics and traces throughout.

This is a conceptual composition, not a mandatory template. A simple chat endpoint needs only ChatClient and ChatModel. RAG and memory are independent capabilities that can be added when the requirements justify them. MCP and tool calling are independent of RAG. The architecture is composable precisely because each capability has a defined boundary.

16. Architectural Design Patterns and Best Practices​

Separation of concerns. Keep the REST layer, business services, prompt construction, model integration, retrieval logic, tool implementations, persistence, and observability in distinct components. Prompt text embedded inside business logic is the most common form of coupling that later resists change.

Provider abstraction with escape hatches. Depend on ChatModel and EmbeddingModel in business code. Where provider-specific capabilities are genuinely required, isolate them in per-provider configuration or a dedicated adapter rather than spreading provider types through the application.

Reusable AI services. Encapsulate shared ChatClient configuration — default system text, advisors, options — in dedicated service or configuration classes. This centralizes behavior that would otherwise be duplicated at call sites.

Modular RAG. Separate ingestion, retrieval, prompt augmentation, and generation. Ingestion is a batch/event-driven pipeline; retrieval is a request-time concern; they fail differently and should be operated differently.

Controlled tool execution. Expose only authorized, validated, narrowly scoped tools. Request-level tool registration exists so that high-risk capabilities are not universally available.

Deliberate conversation management. Design memory storage, conversation identifiers, retention, and eviction as explicit production decisions, not defaults inherited from a development setup.

Production reliability. Timeouts, retry policies aligned with operation idempotency, rate limiting, provider fallback, cost controls, token limits, and concurrency behavior are application responsibilities. The framework provides extension points; it does not make the policy decisions.

Security. Credential management, user authorization, prompt-injection resistance, tool authorization, sensitive-data handling, and trust boundaries around external MCP servers must be designed explicitly. The model is a component that can be influenced by untrusted input, not a trusted principal.

17. Spring AI Architecture vs Direct Provider SDK Integration​

DimensionSpring AIDirect Provider SDK
Abstraction levelApplication framework with portable interfacesProvider client library
Provider integrationMultiple providers behind common abstractionsSingle provider, provider-native types
Model portabilityConfiguration-level switchingCode-level changes
Prompt handlingStructured Prompt / Message modelProvider request objects
Tool callingAdvisor-based loop with portable tool definitionProvider-specific tool schemas
RAG integrationVectorStore, EmbeddingModel, advisorsApplication must implement
Vector store integrationUnified VectorStore interfaceApplication must implement
Spring Boot integrationAuto-configuration and startersManual wiring
Provider-specific featuresAvailable but require escape hatchesFully and directly available
Application complexityLower for multi-provider, RAG, and agentic systemsLower for simple, single-provider use

Spring AI is the appropriate architectural choice when the application is a Spring Boot system, needs more than one provider or anticipates provider changes, requires RAG or tool calling, and values consistent observability and configuration. A direct provider SDK is appropriate when the application is intentionally single-provider, needs deep access to provider-specific APIs, or is a lightweight service where the framework's abstractions add more surface area than value. Both can coexist: provider-specific functionality can be isolated behind an application interface while the rest of the system uses Spring AI abstractions.

18. Spring AI Architecture: Common Misconceptions​

"ChatClient and ChatModel are the same component." ChatClient is the application interaction API; ChatModel is the provider-level model abstraction. ChatClient composes ChatModel and adds the advisor pipeline.

"Spring AI is simply a wrapper around OpenAI APIs." Spring AI supports multiple providers behind portable abstractions, and its OpenAI integration is one implementation among many. The architecture is provider-neutral at the abstraction layer.

"Provider abstraction guarantees identical model behavior." Abstraction normalizes the call interface, not model behavior. Context limits, tool-calling semantics, structured-output enforcement, and output quality remain provider-dependent.

"Advisors and RAG are independent model providers." Advisors are middleware in the request pipeline; RAG is a capability assembled from embedding, vector search, and prompt augmentation. Neither is a provider.

"VectorStore generates embeddings." EmbeddingModel generates embeddings; VectorStore stores and searches them. The two are separate abstractions with a required compatibility constraint between them.

"ChatMemory automatically provides persistent conversation history." ChatMemory provides the logical memory layer; persistence depends on the configured ChatMemoryRepository. An in-memory repository loses history on restart and does not isolate sessions across instances.

"Tool calling means the LLM directly executes application APIs." The model requests a tool call; the application executes it. The model has no direct access to the APIs behind the tools.

"MCP and traditional tool calling are identical." Local tool calling executes in-process functions; MCP consumes capabilities across a protocol boundary from external servers, with different trust and security implications.

"Spring Boot auto-configuration removes the need for architectural design." Auto-configuration removes boilerplate for conventional setups. It does not decide provider abstraction boundaries, memory lifecycle, retrieval design, or security policy.

"Every Spring AI application needs RAG, memory, and agents." These are independent capabilities. A simple chat integration needs only ChatClient and ChatModel; complexity should be added when requirements justify it.

19. Frequently Asked Questions​

What is Spring AI architecture? A layered application architecture: application-facing APIs (ChatClient, Advisors) on top of portable model abstractions (ChatModel, EmbeddingModel, VectorStore), backed by provider and infrastructure integrations, all configured and observed through Spring Boot.

What are the main components of Spring AI? ChatClient, ChatModel, EmbeddingModel, Prompt/Message, ChatResponse, Advisor, VectorStore, Document, ChatMemory, the tool-calling subsystem, and MCP client/server support.

What is the difference between ChatClient and ChatModel? ChatClient is the application-level fluent API that constructs prompts, runs the advisor chain, and extracts responses. ChatModel is the provider-level abstraction that executes the request against a specific model backend.

How does Spring AI connect to different LLM providers? Through provider-specific implementations of ChatModel and EmbeddingModel, selected and configured by Spring Boot auto-configuration or manual bean definition, so that application code depends only on the interfaces.

What role does the Model API play? It defines the portable contracts for model invocation — synchronous and streaming, chat and embedding — and is the seam where provider-specific behavior is isolated from application code.

What are Advisors in Spring AI? Ordered middleware that intercepts the interaction pipeline, can modify or block the prompt, and can post-process the response. Memory, RAG, logging, and tool calling are all implemented as or integrated with advisors.

How does Spring AI implement RAG? As two pipelines: an ingestion ETL path (DocumentReader → transformer/splitter → EmbeddingModel → VectorStore) and a query path that embeds the query, retrieves chunks, augments the prompt, and invokes the model — often encapsulated in a RAG advisor.

How do EmbeddingModel and VectorStore work together? EmbeddingModel converts text to vectors; VectorStore stores vectors and performs similarity search using the same embedding model. The two are separate abstractions with a compatibility requirement between ingestion and query.

How does Spring AI support tool calling? The application defines tools (@Tool, ToolCallback, or FunctionToolCallback); ChatClient auto-registers ToolCallingAdvisor, which loops — model decides, ToolCallingManager executes — until the model returns a response without tool calls. In 2.0 this loop is part of the advisor chain, replacing the per-ChatModel loops of 1.x.

What is the role of MCP in Spring AI? It standardizes connectivity to external tools, resources, and prompts across a client-server protocol boundary, extending the integration scope beyond in-process tools while introducing explicit trust and security considerations.

How does Spring Boot auto-configuration work with Spring AI? Component-specific starters pull per-component auto-configuration artifacts that create the appropriate beans from classpath and configuration properties, with manual configuration available when defaults do not fit.

Can Spring AI be used in enterprise applications? Yes. It inherits Spring Boot's configuration, dependency injection, testing, and observability model, and it provides the abstractions needed to isolate provider-specific behavior.

Is Spring AI suitable for multi-provider AI architectures? Yes. Portable ChatModel and EmbeddingModel abstractions, per-provider configuration, and per-component auto-configuration are designed for multi-provider scenarios, provided provider-specific capabilities are isolated deliberately.

Should developers use Spring AI or a provider-specific Java SDK? Use Spring AI when the application needs Spring integration, provider portability, RAG, tool calling, or consistent observability. Use a provider SDK directly for intentional single-provider systems that need deep provider-specific access or minimal abstraction surface.

Framework

RAG

Vector Databases

Source Code Analysis