Working through this module?
Module 10 · Page 11 of 20 · 11 sections
System Components Deep Dive
Ten building blocks of a production AI system, from client to security, with what each one is responsible for.
End-to-End Architecture
ClientWeb / mobile
Load Balancer
API GatewayAuth, routing
Application ServerOrchestrator
LLM ServiceReasoning & generation
Vector DatabaseRetrieval
External Tools / APIsSearch, DB, etc.
1. Client (Web / Mobile Application)
- Provides the user interface to interact with the system.
- Sends user query, file upload or other input.
- Handles authentication (e.g. JWT token).
- Displays streaming or final response.
- Manages session and chat history.
2. Load Balancer
- Distributes incoming traffic across multiple API servers.
- Improves availability and fault tolerance.
- Performs health checks on backend servers.
- Can be Layer 4 (Nginx) or Layer 7 (e.g. AWS ALB).
- Helps in horizontal scaling.
3. API Gateway
- Single entry point for all client requests.
- Handles authentication and authorization.
- Rate limiting and request validation.
- Routes requests to appropriate services.
- Can transform requests and responses if needed.
- Collects basic metrics and logs.
4. Application Server (Orchestrator)
- Implements business logic and coordinates components.
- Handles user request, session management and context.
- Calls the LLM service, RAG pipeline, external APIs, etc.
- Manages streaming responses to the client.
- Implements error handling, retries and fallbacks.
- Can be scaled horizontally based on load.
5. LLM Service
- Hosts the LLM model (e.g. OpenAI, Llama, Claude).
- Generates responses based on the provided context.
- Supports streaming and non-streaming responses.
- Can be self-hosted or used via a managed API.
- May support multiple models and model switching.
6. Vector Database
- Stores document embeddings for semantic search.
- Used in RAG to retrieve relevant chunks.
- Examples: Pinecone, Qdrant, Weaviate, FAISS.
- Supports similarity search (cosine similarity).
- Stores metadata along with embeddings.
- Helps in retrieving up-to-date and context-relevant data.
Related07RAG System Design
7. External Tools / APIs
- Web search (e.g. SerpAPI, Google Custom Search).
- Enterprise databases (product, user, internal docs).
- File storage (S3, GCS) for documents.
- Other APIs (weather, email, calendar, etc.).
- Used by agents to take actions and get real-time data.
Related09AI Agents Architecture
8. Memory (Conversation and Context)
- Stores chat history and user preferences.
- Can be short-term (in-memory) or long-term (vector DB).
- Helps maintain context across multiple interactions.
- Enables personalized and consistent responses.
- Can store summaries to handle long conversations.
9. Monitoring, Logging and Observability
- Collects logs, metrics and traces from all components.
- Helps in debugging, error detection and performance analysis.
- Tools: Prometheus, Grafana, ELK, OpenTelemetry.
- Monitors latency, token usage, cost and system health.
- Essential for production and scaling.
10. Security and Access Control
- Authenticate users using JWT, OAuth, SSO, etc.
- Authorize access to APIs, data and tools.
- Protect sensitive data (PII).
- Implement rate limiting and abuse prevention.
- Ensure secure communication (HTTPS, encryption).
- Follow responsible AI practices and content filtering.