Working through this module?
Module 15 · Page 16 of 20 · 10 sections
MLOps and LLMOps
Software engineering practice for ML and LLM systems: lifecycle, how LLMOps differs, CI/CD, monitoring and the usual tools.
1. Introduction
- MLOps and LLMOps help in building, deploying and managing machine learning and LLM applications in production.
- They bring software engineering practices like CI/CD, monitoring, versioning and automation into the ML/AI lifecycle.
- MLOps focuses on traditional ML models, while LLMOps is specific to Large Language Models and GenAI applications.
2. MLOps vs LLMOps
| Aspect | MLOps | LLMOps |
|---|---|---|
| Focus | Traditional ML models (classification, regression) | Large language models (LLMs, GenAI) |
| Data | Structured / unstructured | Unstructured text, documents, images, etc. |
| Model size | Usually small to medium | Very large (billions of parameters) |
| Key challenges | Data drift, model drift | Prompt management, context handling, hallucinations |
| Deployment | REST APIs, batch | APIs, vector DB, RAG pipelines, agents |
| Monitoring | Model performance (metrics) | Model + prompt + user feedback + cost |
3. MLOps Architecture
Data CollectionTraining data
Data ProcessingCleaning, feature engineering
Model TrainingTraining, tuning
Model ValidationEvaluation
Model DeploymentREST API, batch
MonitoringMetrics, logs, alerts
RetrainingTrigger on drift
4. Key Components (MLOps)
- Data Pipeline
- Collect, clean and prepare training data.
- Feature Store
- Store and manage features.
- Model Training
- Train and tune the model.
- Model Registry
- Version and store trained models.
- Model Serving
- Deploy the model as an API or batch job.
- Monitoring
- Monitor performance and data drift.
- Retraining
- Automate retraining when needed.
- CI/CD Pipeline
- Automate build, test and deployment.
1
2
3
4
5
6
7
8
5. LLMOps Architecture (Typical RAG-Based)
Data SourcesPDF, DOCX, web, DB, APIs
IngestionChunking, cleaning
Embedding ModelText to vector
Vector DatabasePinecone, Qdrant, etc.
User Query
RetrieverVector search
Prompt ConstructionContext + instructions
LLMGeneration (OpenAI, Llama, etc.)
Response to User
Related07RAG System Design
6. Key Components (LLMOps)
- Data Ingestion
- Load and preprocess documents (PDF, Web, etc.).
- Chunking & Embeddings
- Split text and generate vector embeddings.
- Vector Database
- Store and search embeddings (Pinecone, Qdrant).
- Prompt Management
- Manage and version prompts.
- Model Serving
- Integrate with LLM APIs (OpenAI, Llama, etc.).
- Evaluation
- Measure answer quality (relevance, faithfulness, etc.).
- Monitoring
- Track usage, cost, latency and quality.
- Guardrails
- Handle safety, bias and malicious inputs.
1
2
3
4
5
6
7
8
7. CI/CD Pipeline for MLOps / LLMOps
Code CommitGit push
Build & TestUnit, integration
Model Training
EvaluationAuto + human
DeployCloud / Kubernetes
8. Monitoring and Observability
- Model Performance
- Accuracy, latency, error rate and token usage.
- Data Drift
- Detect changes in input data distribution.
- Model Drift
- Detect degradation in output quality.
- Prompt Monitoring
- Track prompt performance and variations.
- User Feedback
- Collect thumbs up/down and ratings.
- Cost Monitoring
- Track API and infrastructure cost.
- Logging & Tracing
- Use tools like Prometheus, Grafana, ELK and Langfuse.
- Alerts
- Set thresholds and notify on anomalies.
1
2
3
4
5
6
7
8
9. Popular Tools
| Purpose | Tools |
|---|---|
| Data processing | Apache Airflow, Prefect, Dagster |
| Vector database | Pinecone, Qdrant, Weaviate, FAISS |
| Model serving | FastAPI, BentoML, TorchServe, vLLM |
| Experiment tracking | MLflow, Weights & Biases |
| Monitoring | Prometheus, Grafana, Langfuse, Arize, TruLens |
| LLMOps frameworks | LangChain, LlamaIndex, RAGAS, DeepEval |
| CI/CD | GitHub Actions, GitLab CI, Jenkins, ArgoCD |
| Deployment | Docker, Kubernetes, AWS, GCP, Azure |
10. Best Practices
- 1Automate the entire pipeline.
- 2Use versioning for data, models and prompts.
- 3Monitor performance and data drift continuously.
- 4Implement guardrails and safety checks.
- 5Use human evaluation for LLM outputs.
- 6Handle data and model drift.
- 7Keep reproducibility and documentation.
- 8Optimize for latency, cost and scalability.