Spring AI Project Structure

A well-organized project structure is the foundation of every maintainable, scalable Spring AI application. As your system evolves from a simple prototype to a production-grade platform handling retrieval‑augmented generation (RAG), autonomous agents, and multiple model providers, poor package organization leads to tangled dependencies, duplicated configuration, and fragile business logic. This guide provides a proven, production-ready layout that separates concerns, protects your core domain, and makes it easy to swap AI providers or upgrade framework components.

We will start with the essential layered architecture, explore a canonical directory structure, and then dive into each layer: configuration, AI components, RAG modules, multi‑provider support, prompt management, tool calling, agents, enterprise modularization, and testing. By the end, you will be able to design a Spring AI codebase that is clean, extensible, and aligned with Spring Boot conventions.

Typical Spring AI Application Architecture

A Spring AI application shares many architectural characteristics with a standard Spring Boot microservice, but it also introduces AI‑specific concerns such as model clients, advisors, vector stores, and prompt templates. We recommend a layered architecture that isolates these concerns while preserving a clear dependency flow.

Responsibilities:

• Presentation Layer – REST controllers, WebSocket handlers, or UI components. They accept user input and delegate to the application layer, never touching AI logic directly.
• Application Layer – Orchestration and business logic. Services here call ChatClient, compose RAG pipelines, and coordinate domain objects. This layer knows what to do but not how the AI internals work.
• AI Layer – Core AI building blocks: ChatClient, ChatModel, PromptTemplate, advisors, memory, tool registrations, and streaming. This layer encapsulates the Spring AI framework specifics.
• Domain Layer – Domain entities, value objects, and DTOs. These are plain Java objects with no framework dependencies, ensuring they remain portable and testable.
• Infrastructure Layer – Configuration, provider beans, external service clients, and cross‑cutting concerns like logging and security. This is where auto‑configuration and manual bean definitions reside.
• Data Layer – Vector stores (e.g., PgVectorStore), document repositories, and any other persistence mechanisms.

This layered approach mirrors classic Spring patterns and allows you to test each layer in isolation, swap AI providers without touching business logic, and extend functionality (e.g., adding agents) in a predictable manner.

Recommended Project Directory Structure

The following directory layout reflects the architecture above. It is suitable for a single-module Maven/Gradle project that can later be split into multiple modules when the codebase grows.

src/
└── main/
    ├── java/
    │   └── com/example/ai/
    │       ├── config/
    │       │   ├── AiConfig.java
    │       │   └── ProviderConfig.java
    │       ├── controller/
    │       │   └── ChatController.java
    │       ├── service/
    │       │   ├── ChatService.java
    │       │   └── KnowledgeBaseService.java
    │       ├── ai/
    │       │   ├── chat/
    │       │   │   └── ChatClientProvider.java
    │       │   ├── prompt/
    │       │   │   ├── PromptTemplateManager.java
    │       │   │   └── templates/
    │       │   ├── memory/
    │       │   │   └── ChatMemoryConfig.java
    │       │   ├── tools/
    │       │   │   ├── ToolDefinitions.java
    │       │   │   └── WeatherTool.java
    │       │   ├── advisors/
    │       │   │   └── LoggingAdvisor.java
    │       │   └── providers/
    │       │       ├── openai/
    │       │       ├── ollama/
    │       │       └── azure/
    │       ├── rag/
    │       │   ├── ingestion/
    │       │   ├── chunking/
    │       │   ├── embedding/
    │       │   ├── retrieval/
    │       │   └── reranking/
    │       ├── vectorstore/
    │       │   └── VectorStoreProvider.java
    │       ├── agent/
    │       │   ├── AgentService.java
    │       │   └── plans/
    │       ├── workflow/
    │       │   └── WorkflowEngine.java
    │       ├── model/
    │       │   ├── ChatMessage.java
    │       │   ├── DocumentChunk.java
    │       │   └── AgentState.java
    │       ├── dto/
    │       │   ├── ChatRequest.java
    │       │   └── ChatResponse.java
    │       ├── exception/
    │       │   └── AiExceptionHandler.java
    │       ├── repository/
    │       │   └── DocumentRepository.java
    │       └── util/
    │           └── PromptUtils.java
    └── resources/
        ├── application.yml
        ├── application-dev.yml
        ├── prompts/
        │   ├── system-qa.st
        │   └── summarizer.st
        ├── templates/
        └── documents/

Purpose of each directory:

• config/ – All Spring configuration classes (e.g., @Configuration, @Bean definitions) that wire AI components together.
• controller/ – REST endpoints that receive user requests and delegate to services.
• service/ – Business logic and orchestration. These classes should be framework‑agnostic and depend on interfaces rather than concrete AI implementations.
• ai/ – The heart of your AI logic, divided by concern:
  • chat/ – Chat client instantiation and helper methods.
  • prompt/ – Prompt template management and loading from resource files.
  • memory/ – Custom chat memory implementations or configuration.
  • tools/ – @Tool-annotated methods and tool registries.
  • advisors/ – Custom advisors for logging, security, or context enrichment.
  • providers/ – Provider‑specific adapters or configuration, if you need fine‑grained control beyond auto‑configuration.
• rag/ – Retrieval‑augmented generation pipeline components, organized by stage (ingestion, chunking, embedding, retrieval, reranking).
• vectorstore/ – Vector store abstractions and provider‑specific beans.
• agent/ – Agent implementations, planning logic, and execution loops.
• workflow/ – Workflow engine or state machine implementations (for advanced orchestration).
• model/ – Domain models and entities used across the application.
• dto/ – Data Transfer Objects for API requests/responses.
• exception/ – Global AI exception handling (e.g., @ControllerAdvice).
• repository/ – Data access for relational or NoSQL databases (excluding vector stores, which live in vectorstore/).
• util/ – Static utility functions (prompt helpers, token estimators, etc.).

Resource directories:

• prompts/ – .st (StringTemplate) files or plain text templates for system and user prompts.
• templates/ – Thymeleaf or other view templates (if your application serves a UI).
• documents/ – Sample documents used for testing or seeding local vector stores.

This layout scales naturally. When you later decide to split the application into multiple Maven modules, you can lift entire packages (e.g., ai/, rag/) into their own modules with minimal refactoring.

Configuration Layer

Organize configuration classes to mirror the features they set up.

@Configuration
public class AiConfig {

    @Bean
    public ChatClient chatClient(ChatClient.Builder builder) {
        return builder
                .defaultAdvisors(new LoggingAdvisor())
                .build();
    }
}

If you support multiple providers, create separate configuration classes or use Spring profiles:

• OpenAiConfig.java
• OllamaConfig.java
• AzureOpenAiConfig.java

Each class is annotated with @Profile("openai") and defines the appropriate ChatModel bean. Centralize prompt template loading in a PromptTemplateConfig that scans the prompts/ directory and exposes a PromptTemplateManager.

Environment‑specific settings are managed via application-{profile}.yml. For example, application-dev.yml might use an Ollama local model, while application-prod.yml points to Azure OpenAI with higher timeouts.

AI Components Organization

The ai/ package contains all Spring AI‑specific components, neatly separated by role.

ChatClient

Create a dedicated provider class that constructs and configures the ChatClient bean. This centralizes the default settings (temperature, model, advisors) and makes it easy to swap configurations per environment.

ChatModel

Wrap low‑level ChatModel access behind a simple service interface if you need to support provider‑agnostic operations. For most use cases, ChatClient is sufficient.

PromptTemplate

Externalise prompt templates as .st files under resources/prompts/. Load them with @Value or a custom PromptTemplateManager. This decouples prompt engineering from code and enables versioning, A/B testing, and localization.

Advisors

Place custom advisors in ai/advisors/. A logging advisor that records prompts and responses, or a security advisor that sanitizes inputs, are common examples. Advisors are then registered on the ChatClient builder.

Memory

Chat memory implementations (e.g., in‑memory, JDBC‑backed) go into ai/memory/. The configuration class selects the appropriate ChatMemory bean based on the active profile.

Tool Calling

Tool definitions are kept in ai/tools/. Each tool is a separate class annotated with @Tool. A tool registry can be created to collect and expose them to the advisor chain.

Structured Output

Classes that handle JSON‑to‑object conversion can reside in ai/chat/ or be extracted into a dedicated ai/structured/ package if the application relies heavily on typed responses.

Streaming

Streaming endpoints might require special Flux‑based controllers. Keep the streaming‑specific logic close to the chat client configuration; the controller merely subscribes.

Organizing RAG Modules

A RAG pipeline consists of several sequential stages. The rag/ package mirrors this flow:

rag/
├── ingestion/
│   └── PdfIngestionService.java
├── chunking/
│   └── TokenChunker.java
├── embedding/
│   └── EmbeddingService.java
├── retrieval/
│   └── VectorStoreRetriever.java
└── reranking/
    └── CohereReranker.java

• ingestion/ – Document readers and preprocessing (PDF, HTML, plain text).
• chunking/ – Custom DocumentTransformer implementations for splitting text.
• embedding/ – Thin wrapper around EmbeddingModel that handles batching and error handling.
• retrieval/ – Logic that queries the vector store, applies metadata filters, and returns candidate documents.
• reranking/ – Integration with external re‑ranking models or cross‑encoders.

Each stage is a separate Spring bean, making it easy to replace or mock during testing. The orchestration is typically done in a service class (e.g., KnowledgeBaseService) that calls these beans sequentially.

Managing Multiple AI Providers

Supporting multiple providers (OpenAI, Azure, Ollama, etc.) is a first‑class requirement for many enterprises. The project structure should reflect this without creating duplication.

Strategy:

1. Use Spring profiles to select the active provider.
2. Define a ChatModel bean per profile in ai/providers/.
3. Inject ChatModel (or ChatClient) into your services without ever referencing a concrete implementation.

Example package for provider‑specific code:

ai/
└── providers/
    ├── openai/
    │   └── OpenAiConfig.java
    ├── azure/
    │   └── AzureOpenAiConfig.java
    └── ollama/
        └── OllamaConfig.java

If you need to access provider‑specific features (e.g., custom headers), encapsulate that logic inside the provider package and expose it through a common interface. Never leak provider‑specific types into the service layer.

Prompt Management

Prompt templates should be treated as external resources, not as inline strings. Place them under resources/prompts/ with descriptive names:

• system-qa.st
• system-summarizer.st
• user-document-qa.st

Load them via a PromptTemplateManager:

@Component
public class PromptTemplateManager {

    private final Map<String, PromptTemplate> templates = new HashMap<>();

    public PromptTemplateManager(ResourceLoader resourceLoader) {
        // scan prompts/ folder and load .st files
    }

    public PromptTemplate get(String name) {
        return templates.get(name);
    }
}

Version your prompts by keeping them in the same repository as your code (Git). For advanced scenarios, store prompts in a database or a dedicated config server, but always provide a local fallback for development.

Tool Calling Organization

Tools are regular Spring beans annotated with @Tool. Group them by domain in ai/tools/:

ai/tools/
├── WeatherTool.java
├── DatabaseLookupTool.java
├── CalculatorTool.java
└── ToolRegistryConfig.java

The ToolRegistryConfig scans the application context and collects all @Tool beans. This registry is then passed to the advisor that enables tool calling. By isolating tools in their own package, you make it easy to add, remove, or test them in isolation.

Each tool should validate its inputs and return structured results. Error handling (e.g., wrapping exceptions in a user‑friendly message) can be centralized through an ToolExceptionHandler.

AI Agent Organization

Agents add complexity by looping, planning, and maintaining state. Organize them under agent/:

agent/
├── AgentService.java
├── plans/
│   ├── PlanGenerator.java
│   └── PlanStep.java
├── execution/
│   └── AgentExecutor.java
└── memory/
    └── AgentMemoryStore.java

• AgentService – Public API for invoking an agent.
• plans/ – Logic that generates a plan (step‑by‑step instructions) from a user goal.
• execution/ – Loop that iterates through plan steps, calls tools, and updates memory.
• memory/ – Persistence for agent state across turns (can reuse ChatMemory or a dedicated store).

Agents often interact with MCP (Model Context Protocol). MCP client/server implementations can live in a separate mcp/ package under agent/ or at the top level if shared by multiple agent implementations.

Enterprise Project Structure

For large, multi‑team Spring AI projects, a single module becomes unwieldy. Split the application into Maven modules that align with the architectural layers:

• ai-core – Core interfaces (ChatClient, ChatModel, VectorStore, DocumentReader) and common domain objects.
• ai-provider-openai – OpenAI‑specific adapter (implements core interfaces).
• ai-provider-azure – Azure‑specific adapter.
• ai-rag – RAG pipeline components (ingestion, chunking, retrieval).
• ai-agent – Agent framework, planning, execution.
• ai-api – REST controllers and DTOs.
• ai-web – Web application (Thymeleaf, React) if needed.
• ai-monitoring – Observability integrations (metrics, tracing).

Each module has its own pom.xml (or build.gradle) and depends on ai-core. The ai-api module wires everything together and depends on the provider and RAG modules through runtime scope, enabling easy provider switching.

This modular layout enforces strict dependency boundaries and allows different teams to work on different modules in parallel. It also simplifies testing and deployment because you can build and deploy only the modules that changed.

Testing Structure

Mirror the production package structure under src/test/java/. This makes it easy to locate tests and ensures coverage.

src/test/java/com/example/ai/
├── ai/
│   ├── chat/
│   ├── tools/
│   └── advisors/
├── service/
├── controller/
└── rag/

Testing best practices:

• Unit tests – Use JUnit 5 and Mockito to test individual AI components. Mock ChatModel and VectorStore to test advisor logic or tool execution without making real API calls.
• Integration tests – Use @SpringBootTest with profiles that point to local Ollama or an embedded vector store (e.g., SimpleVectorStore). Verify end‑to‑end flows.
• Mock AI providers – Create a test configuration that replaces the ChatModel bean with a MockChatModel that returns predefined responses. This isolates your tests from external services.
• Test prompts – Store test‑specific prompts under src/test/resources/prompts/ to validate template processing.
• Performance tests – Add Gatling or JMeter scripts in a separate src/test/perf/ directory to measure token usage and response latency.

Common Mistakes

• Mixing business logic with AI logic – Services that build prompts, call ChatClient, and process results become hard to test. Delegate AI‑specific operations to dedicated AI components.
• Hardcoded prompts – Strings scattered across controllers and services make prompt tuning a nightmare. Externalize all prompts.
• Poor package organization – A single ai package with 50 files obscures dependencies. Split by feature and layer from the start.
• Tight provider coupling – Importing OpenAiChatModel directly in a service destroys portability. Always depend on the ChatModel interface.
• Duplicated prompt templates – Copy‑pasting prompts across services leads to inconsistencies. Centralize them in prompts/.
• Missing configuration abstraction – Every component that needs an API key shouldn’t read it from application.yml directly. Use a configuration class that exposes typed properties.
• Lack of separation of concerns – A single AIService that handles chat, RAG, and tools will quickly become unmaintainable. Break it into smaller, focused services.

Best Practices

• Keep AI logic isolated in the ai/ package, away from business services.
• Prefer interfaces (ChatModel, VectorStore, PromptTemplate) over concrete implementations. This enables testing and provider switching.
• Externalize prompts into resource files and load them with a dedicated manager.
• Use dependency injection extensively. Let Spring wire together the advisor chain, tools, and providers.
• Design for provider switching from day one. Even if you start with only OpenAI, hide it behind the abstraction.
• Organize reusable components (advisors, tools, prompt templates) in a way that they can be shared across multiple services or modules.
• Maintain clean package boundaries: service/ depends on ai/, not the other way around. ai/ depends on interfaces, not on concrete provider classes.
• Follow Spring Boot conventions: use @Configuration classes, application.yml for configuration, and @Component scanning.

What's Next

With a solid project structure in place, you are ready to deepen your understanding of individual Spring AI components:

• Spring AI Architecture – Understand the framework’s internal design.
• ChatClient Guide – Master the fluent API for building requests.
• ChatModel & Providers – Learn how to configure and switch AI backends.
• Prompt Engineering – Design effective prompts with templates.
• Tool Calling – Let the model invoke your Java methods.
• RAG Overview – Dive into retrieval‑augmented generation.
• Enterprise AI Patterns – Production security, observability, and deployment.

Key Takeaways

• A clean project structure separates presentation, application, AI, domain, infrastructure, and data layers.
• The ai/ package organizes all Spring AI‑specific code: chat clients, prompts, tools, advisors, memory.
• RAG pipelines benefit from a dedicated rag/ package with sub‑packages for each stage.
• Support multiple AI providers by hiding them behind profiles and the ChatModel interface.
• Externalize and version prompt templates; never embed them directly in code.
• For large projects, split into Maven modules that enforce architectural boundaries.
• Mirror the production structure in tests to keep them maintainable.
• Avoid common pitfalls like provider coupling, hardcoded prompts, and mixing concerns.