How to read this list
The practices below are ordered roughly by impact — how much production pain they prevent per unit of effort. The foundation (versioning, reproducible environments, experiment tracking) comes first because everything above it depends on it. Then a reliability layer that keeps a live model trustworthy, a scale layer for when you have many models or high traffic, and a governance cap for cost, security, and compliance. You don't need all 14 on day one — adopt them as your stakes and scale grow. Each item says what it is, when it matters most, and gives a concrete example of the failure it prevents.
1. Version everything — code, data, models, and config
Foundation
Put every input that shapes a model under version control: source code, the exact dataset, the trained model artifact, and the config/hyperparameters. Tools: Git for code, DVC or lakeFS for data, a model registry for artifacts.
When it matters most: Most valuable the moment more than one person touches the system, or when you need to reproduce or roll back a model that's been in production for months.
Example: A fraud model starts flagging good transactions. Because the training data snapshot and config were versioned, you diff last week's model against this week's, spot a corrupted feature in the new data pull, and roll back in minutes — instead of guessing for days.
2. Monitor models in production (not just infra)
Reliability
Track model-level signals continuously: prediction distribution, input-data drift, feature quality, latency, and — where you can get labels — live accuracy. Alert on drift, not just on 500s.
When it matters most: Critical for any model facing a changing world: user behavior, markets, seasonality, adversaries. Uptime dashboards will say 'healthy' while the model quietly rots.
Example: A demand-forecasting model looks fine on Grafana (CPU, latency normal) but a drift alert fires: input distributions shifted after a pricing change. You catch degrading forecasts a week before finance would have noticed in the numbers.
3. Automate the training-to-deployment pipeline (CI/CD for ML)
Reliability
Make training, testing, packaging, and deployment a repeatable pipeline triggered by code or data changes — not a sequence of manual notebook runs. Tools: GitHub Actions, GitLab CI, Kubeflow Pipelines, or Airflow.
When it matters most: Pays off as soon as you retrain regularly or have more than a couple of models. Manual deploys don't scale and are where most production incidents are born.
Example: A new labeled batch lands weekly. The pipeline auto-trains, runs evaluation gates, and — only if metrics pass — promotes the model to staging for a one-click prod release. What used to be a nervous half-day is now a green checkmark.
4. Make environments reproducible
Foundation
Pin dependencies and containerize training and serving so the same code produces the same result anywhere. Tools: Docker, uv/Poetry lockfiles, and identical base images across train and serve.
When it matters most: Essential when 'works on my machine' meets a different GPU driver, CUDA version, or library minor-bump in production. Also the backbone of reproducibility for audits.
Example: A model trains fine locally but gives different outputs in prod. The cause: a silently upgraded library. A pinned, containerized environment eliminates the whole class of 'it changed under me' bugs.
5. Track experiments systematically
Foundation
Log every run's parameters, metrics, datasets, and artifacts so results are comparable and searchable. Tools: MLflow, Weights & Biases.
When it matters most: Invaluable during active model development and whenever someone asks 'which run produced the model we shipped?' six weeks later.
Example: Three engineers try 40 configurations for a recommender. Because every run is tracked, you sort by offline NDCG, find the winner, and know exactly which data and hyperparameters produced it — no lost notebooks, no re-running.
6. Validate data at the gates (train and serve)
Reliability
Automatically check schema, ranges, nulls, and distributions on data entering training and inference. Reject or quarantine bad data before it reaches the model. Tools: Great Expectations, Pandera, TFX Data Validation.
When it matters most: Highest value with upstream data you don't control — third-party feeds, event streams, other teams' tables — where a silent format change can poison a model.
Example: An upstream service starts sending prices in cents instead of dollars. A serving-time validation rule catches the 100x range violation and fails safe, instead of the model confidently producing garbage predictions for a day.
7. Put automated evaluation gates before every deploy
Reliability
Block promotion unless a candidate model clears thresholds: offline metrics, regression tests against a fixed benchmark set, fairness checks, and slice-level performance. No green, no ship.
When it matters most: Crucial once retraining is automated — otherwise automation happily ships a worse model. Also protects vulnerable slices from silent regressions.
Example: A retrained model improves overall accuracy but the gate flags a 6-point drop on a key customer segment. Deployment is blocked automatically, and you fix the imbalance before any user is affected.
8. Use a model registry with staged promotion
Reliability
Register models as versioned artifacts and move them through explicit stages (dev → staging → production) with approvals and metadata. Tools: MLflow Model Registry, SageMaker Model Registry.
When it matters most: Important as soon as you have multiple models, multiple environments, or any compliance need to know exactly what's serving and who approved it.
Example: An incident asks 'what model was live at 2pm Tuesday?' The registry answers instantly — version, training run, approver — and gives you a one-click rollback to the previous production stage.
9. Roll out safely — canary, shadow, and instant rollback
Scale
Never flip 100% of traffic to a new model. Use shadow mode (score live traffic without acting), canaries (small % first), or blue-green, and keep rollback one command away.
When it matters most: Essential for high-traffic or high-stakes models where a bad deploy is expensive. Shadow mode is especially useful before you trust a model at all.
Example: A new pricing model runs in shadow for a week: you compare its prices to production without charging anyone. It looks great in aggregate but over-prices one region — caught with zero customer impact before the real rollout.
10. Kill train-serve skew with consistent features
Scale
Compute features the same way in training and serving — ideally from one shared pipeline or a feature store — so the model sees at inference what it saw during training.
When it matters most: A top cause of 'great offline, bad online' models. Most dangerous when training features are computed in batch SQL and serving features in application code.
Example: A churn model uses a 30-day rolling average computed one way in the training warehouse and slightly differently in the live app. A feature store makes both read the identical definition, and online accuracy jumps to match offline.
11. Capture lineage and end-to-end observability
Scale
Trace every prediction back through the model version, features, and source data that produced it, and keep logs/traces for debugging. Tools: OpenLineage, tracing on your serving layer.
When it matters most: Indispensable during incidents and audits — when you must explain why a specific prediction happened, or which downstream models a bad dataset touched.
Example: A customer disputes an automated decision. Lineage lets you reconstruct the exact model version and feature values used for their case in minutes, satisfying both the customer and a compliance reviewer.
12. Close the loop — feedback and scheduled retraining
Scale
Capture ground-truth outcomes and feed them back into retraining on a cadence (or triggered by drift), with the same gates as any other deploy. Automate the loop, don't retrain by heroics.
When it matters most: Vital for models in fast-moving domains (recommendations, fraud, forecasting) where last quarter's model is already out of date.
Example: A fraud model retrains nightly on freshly labeled outcomes and auto-promotes only if it beats the incumbent on a held-out set — staying ahead of adversaries without an engineer babysitting it.
13. Manage cost and compute deliberately
Governance
Right-size training and serving, use spot/preemptible instances for interruptible jobs, autoscale inference, cache and batch where possible, and quantize models that don't need full precision.
When it matters most: Matters most at scale, on GPUs, or with LLM inference — where a naive setup can 10x the bill for the same output.
Example: An LLM feature serves every request at full precision on always-on GPUs. Quantizing the model and autoscaling to demand cuts the monthly inference bill by 60% with no measurable quality loss.
14. Bake in governance, security, and compliance
Governance
Control access to data and models, log who did what, protect PII, document models (model cards), and keep an audit trail. Treat models and their data as first-class security assets.
When it matters most: Non-negotiable in regulated domains (finance, healthcare) and increasingly expected everywhere under regimes like the EU AI Act.
Example: An auditor asks how a credit model handles protected attributes. Because model cards, access logs, and evaluation slices were maintained from day one, you produce the evidence the same afternoon instead of scrambling for weeks.
Putting it together
If you do nothing else, do the foundation: building anything non-trivial on ML — including modern LLM and agent systems — falls apart without reproducibility and tracking. Layer on monitoring and evaluation gates before you automate retraining, or you'll automate your way into shipping worse models. Add safe rollouts, feature consistency, and lineage as traffic and stakes rise. And treat governance not as paperwork but as the thing that lets you move fast without fear when the auditor — or the incident — arrives.
Frequently Asked Questions
What is MLOps in simple terms?
MLOps is DevOps for machine learning: the practices and tooling that take a model from a notebook to a reliable, monitored, repeatable production system — covering versioning, automated pipelines, deployment, monitoring, and governance.
Where should a small team start?
Start at the foundation: version your code, data, and models, make environments reproducible, and track experiments. Those three unlock everything above them and prevent the most painful failures — reproducing or rolling back a model — with the least effort.
How is MLOps different from DevOps?
DevOps ships code; MLOps ships code plus data plus a trained model, all three of which change independently. That's why MLOps adds data validation, model monitoring for drift, experiment tracking, and evaluation gates that plain DevOps doesn't need.
Do I need all 14 practices to have good MLOps?
No. Adopt them by impact and by your situation. A low-stakes internal model may need only the foundation and basic monitoring; a regulated, high-traffic model needs the full stack including safe rollouts, lineage, and governance.