Working through this module?0 of 19 done
Module 05 · Page 6 of 20 · 8 sections

Model Serving and Inference Architecture

How a trained model is exposed to users: serving styles, the request flow, popular tools and the trade-offs to weigh.

04Model Training Pipeline06LLM System Design10System Components Deep Dive11Reliability, Fault Tolerance and High Availability
5.1

What is Model Serving?

  • The process of deploying a trained model and making it available for real-time or batch inference.
  • It handles incoming requests, runs inference and returns the response to the user or application.
  • It includes infrastructure, APIs, scaling, monitoring and security.
5.2

Why is it Important?

  • Brings ML/AI models to real users.
  • Handles high traffic with low latency.
  • Ensures reliability, scalability and availability.
  • Supports versioning, A/B testing and rollback.
  • Enables cost-efficient and optimized inference.
  • Monitors model performance and data drift in production.
5.3

Typical Inference Architecture

Users / ClientsWeb / mobile
Load Balancer
API GatewayAuth, rate limiting, routing
Inference ServerLoad model, run inference, auto-scale
Model StorageRegistry, versioning, artifacts
MonitoringLogs, metrics, alerts
5.4

Types of Model Serving

1
Real-time Inference
Low latency response (milliseconds). Used for chatbots and recommendations.
2
Batch Inference
Process large data offline. Used for data enrichment and reports.
3
Online Serving
Model is always available in memory. Used for real-time applications.
4
Serverless Serving
Pay per request with auto scaling. Used for low or variable traffic.
5
Edge Inference
Run models on edge devices (mobile, IoT). Used for low latency and privacy-sensitive use cases.
5.5

Inference Flow (Step-by-Step)

  1. 1Client sends a request (text, image, etc.) to the API.
  2. 2API Gateway validates the request (auth, rate limit).
  3. 3Request is routed to the inference server.
  4. 4Inference server loads the model (if not already loaded).
  5. 5Preprocess the input data.
  6. 6Run inference using the model.
  7. 7Postprocess the output (decode, format).
  8. 8Return the response to the client.
5.6

Popular Model Serving Tools

ToolBest for
TensorFlow ServingTensorFlow models
TorchServePyTorch models
FastAPILightweight Python API server
BentoMLUnified model serving framework
NVIDIA TritonHigh performance multi-model serving
Seldon CoreKubernetes-based model serving
Hugging Face Inference EndpointsLLM serving
5.7

Key Considerations

1
Latency
Optimize model and infrastructure.
2
Throughput
Handle a high number of requests.
3
Scalability
Auto scaling with load.
4
Model Versioning
Support multiple model versions.
5
Monitoring
Track latency, errors and drift.
6
Security
Authentication, authorization and data privacy.
7
Cost Optimization
Use the right instance type and batching.
Related11Reliability, Fault Tolerance and High Availability12Security, Privacy and Responsible AI13Monitoring, Logging and Observability
5.8

Real-world Examples

1
ChatGPT
Real-time LLM inference at scale.
2
Netflix
Recommendation model serving.
3
Uber
ETA prediction model serving.
4
Google
Search ranking model serving.
5
E-commerce
Product recommendation.
6
Healthcare
Medical image analysis inference.