Contents
Figure 6: Drift monitoring catches silent degradation before stakeholders notice — the difference between proactive fixes and reactive crises
Here's a story worth sitting with before anything else: six months ago, a customer churn model was the hero of the retention team — 85% precision flagging at-risk users, marketing acting on every alert. Then, slowly, almost imperceptibly, the alerts started feeling off. Fewer high-risk customers got caught, and the ones that were flagged didn't actually churn. The model didn't break. Nothing crashed. No error fired. The world around it changed, and nobody was watching for that.
This is genuinely the payoff article for the entire MLOps and data-engineering arc running through this series — recall the CI/CD article's Friday-afternoon regression, caught (or not) by a quality gate at deployment time. This article is about the much longer, quieter failure that happens after deployment, when a model that passed every gate perfectly on day one starts silently degrading on day two hundred. This is genuinely the #1 reason ML projects fail in production — not a dramatic crash, a slow, silent bleed nobody notices until a stakeholder complains.
By the end of this guide, you'll understand the distinct types of drift, build a working detection pipeline with Evidently AI, and know exactly how to turn a detected drift signal into an actual action rather than just another dashboard nobody checks. IMO, "the silent killer" framing genuinely earns its drama here — most teams truly don't find out until someone outside the ML team notices first :)
The Three (Genuinely Distinct) Types of Drift
Model drift isn't one phenomenon — it's at least three, and conflating them leads to monitoring the wrong thing.
Data drift — the statistical distribution of input features changes over time. A fraud detection model trained on 2024 transaction patterns suddenly receiving payment data from a platform that barely existed during training is the textbook example — the world simply generated new kinds of inputs the model never saw.
Concept drift — the relationship between features and the target changes, even if the inputs themselves look statistically similar. The same inputs now map to a genuinely different correct output — markets shift, fraud patterns evolve, user behavior changes in ways that redefine what "risky" or "normal" actually means.
Prediction drift — the distribution of the model's own outputs shifts, which can be an early symptom of either of the above, and is genuinely useful to monitor even when you don't have fresh ground-truth labels to check against directly.
Most monitoring setups default to watching feature drift specifically because it's the easiest to detect and often the first visible symptom — but a genuinely robust setup watches all three, since concept drift in particular can occur with no visible change in the input distribution at all.
The Four-Layer Monitoring Strategy
A complete monitoring strategy covers layers beyond just "is drift happening," stacking from upstream data quality through to actual business impact.
Data quality monitoring — catches upstream problems before they ever reach the model: null rates, out-of-range values, schema changes. Recall the CI/CD article's data validation step; this is that same concern, now running continuously in production rather than just at training time.
Feature/input drift monitoring — detects when input distributions shift meaningfully from the training baseline.
Prediction/output drift monitoring — watches the distribution of what the model actually produces, catching behavioral shifts even before you have fresh labels to confirm accuracy has genuinely dropped.
Performance and business metric correlation — tracks actual accuracy metrics (when labels are available) and, genuinely more importantly, whether drift is translating into real downstream business impact — because a model can technically drift statistically while performance stays fine, and the reverse is also true.
Programs that monitor only one or two of these layers have predictable, genuinely avoidable blind spots — each layer catches a distinct class of failure the others miss entirely.
Building a Drift Detection Pipeline With Evidently AI
Recall Evidently AI's recommendation from the MLOps beginners guide — here's what actually implementing it looks like.
pip install evidentlyimport pandas as pd
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, DataQualityPreset
reference_data = pd.read_parquet("data/training_snapshot.parquet")
current_data = pd.read_parquet("data/last_24h_production.parquet")
report = Report(metrics=[DataDriftPreset(), DataQualityPreset()])
report.run(reference_data=reference_data, current_data=current_data)
report.save_html("drift_report.html")Your reference_data should genuinely be the exact sample used to train the model — ideally the same DVC-versioned dataset from the previous article, so "what did the model actually learn from" and "what am I comparing against" are unambiguous, not approximated. DataDriftPreset automatically runs statistical tests — PSI (Population Stability Index) and Kolmogorov-Smirnov tests — across every feature, producing a visual HTML report you can genuinely open and read without writing any custom statistics yourself.
Reading the Report: What Actually Matters
Open the report and sort by drift score, descending. A handful of features showing meaningful drift while most stay stable is a completely different situation from broad, pipeline-wide drift across nearly everything — the first suggests a specific upstream cause worth chasing down; the second suggests something more systemic, possibly a pipeline bug rather than genuine real-world change.
Drift detection tells you what is drifting, not why. Is it seasonal? A data pipeline bug upstream? A genuine real-world behavioral shift? That diagnosis step is still yours to do — the tool flags the signal, it doesn't hand you the root cause.
From Detection to Action: Alerts, Not Just Reports
Here's the part genuinely worth internalizing as non-negotiable: don't just generate reports — trigger alerts automatically. A beautiful HTML report nobody opens is functionally identical to no monitoring at all.
import requests
drift_detected = report.as_dict()["metrics"][0]["result"]["dataset_drift"]
if drift_detected:
requests.post(
SLACK_WEBHOOK_URL,
json={"text": "Data drift detected in production model — review drift_report.html"}
)Connect this to Slack, email, or PagerDuty so your team is notified the moment drift appears, not the moment a stakeholder happens to notice degraded results and asks about it — recall the churn model story that opened this article, where the alert genuinely should have fired months before a human noticed anything felt "off."
Scheduling: This Needs to Run Automatically, Not "When Someone Remembers"
Drift detection should run on a schedule, not manually. Recall the Airflow article directly here — the exact same orchestration pattern applies.
from airflow.decorators import dag, task
from datetime import datetime
@dag(schedule="@daily", start_date=datetime(2026, 1, 1))
def drift_monitoring_pipeline():
@task
def run_drift_check():
# the Evidently report logic above
...
run_drift_check()A genuinely useful rule of thumb on frequency: run drift detection daily for revenue-critical models, weekly for others. The cost of a single missed drift event vastly outweighs the cost of running the check regularly — this isn't a place to economize on compute to save a few dollars a month.
The Retraining Decision: Drift Alone Isn't Automatically the Answer
Detecting drift doesn't automatically mean "retrain immediately." If drift is real and significant, retrain on newer labeled data — but genuinely confirm it's real and significant first, not just statistically detectable at some threshold.
Seasonal drift might resolve on its own and not warrant a retrain at all — a retail model seeing different purchasing patterns every December isn't necessarily broken.
A data pipeline bug upstream should be fixed at the source, not papered over by retraining a model on corrupted inputs.
A genuine real-world behavioral shift is exactly the case that warrants retraining — and this is precisely where the CI/CD article's Continuous Training concept and this article's drift detection connect directly: drift detection is genuinely the trigger condition CT pipelines should react to.
Choosing a Monitoring Tool at Your Actual Scale
Evidently AI — genuinely the right open-source starting point for most teams; its core strength is simplicity — HTML reports requiring no external platform, easy to share, easy to get running in an afternoon.
WhyLabs — specifically architected for genuinely massive throughput via a profiling-based approach, worth considering once you're dealing with very high daily prediction volumes.
Arize AI and Fiddler — commercial platforms recommending sampling above roughly 100M daily predictions; this works, but genuinely introduces some uncertainty into drift detection at that extreme scale.
For lower-throughput applications — fraud detection, credit decisions, medical diagnosis models processing thousands to millions of predictions daily — any of these tools handles scale comfortably; optimize your choice for other criteria instead, like explainability needs or existing infrastructure fit.
Fiddler specifically provides the deepest explainability integration, worth prioritizing if your use case involves high-stakes decisions — loan approvals, medical diagnoses, hiring — where you genuinely need to explain why a model made a specific call, not just detect that its behavior shifted.
A Genuinely Important Distinction: Statistical Drift vs. Behavioral Assurance
Worth naming directly for anyone in a regulated industry: most drift detection tools track data distribution shifts, not whether your AI is still behaving correctly according to policy or regulation — and these are genuinely complementary concerns, not substitutes for each other.
Data drift monitoring tells your data science team that production inputs are diverging from training data, which may warrant retraining — this is the entire focus of everything covered above.
Behavioral drift/assurance tells a compliance team that AI outputs are changing in ways that may violate regulatory requirements — a genuinely different question, answered for a different audience, with different evidentiary requirements.
Insurance, banking, and healthcare face accelerating regulatory pressure specifically here — the NAIC Model Bulletin has been adopted in over half of US states as of early 2026, with formal evaluation tooling entering pilot examinations. If you're in a regulated industry, statistical drift monitoring alone genuinely doesn't satisfy this requirement — treat it as one layer of a broader governance posture, not the whole thing.
Extending This to LLMs
Recall the RAG and local LLM arcs from earlier in this series — drift monitoring genuinely extends to generative models too, with some real adaptation.
Evidently has extended its coverage to LLM outputs specifically, supporting text generation monitoring alongside its original tabular data drift reports.
Giskard is a dedicated open-source framework for LLM-specific test suites — covering drift, factual consistency, safety, and bias — genuinely strong specifically for comparing two model versions before promoting a new deployment, connecting directly to the CI/CD article's quality-gate concept.
The realistic cost here is genuinely modest: a monitoring stack built on open-source tooling (Evidently, Prometheus, Grafana) plus a managed LLM-evaluation platform is operationalizable by a small team in a few weeks of setup, with maybe a few hours a month of ongoing maintenance. The real risk is flying blind with a production LLM, not the cost of building this.
A Practical Starting Sequence
If you're implementing this for the first time rather than retrofitting an existing production model, here's a sensible order.
Start with data quality checks — nulls, range violations, schema changes — the cheapest, highest-signal layer to implement first.
Add a weekly drift check using Evidently's DataDriftPreset, comparing production data against your DVC-versioned training reference.
Monitor your prediction distribution directly, even before you have fresh ground-truth labels available to check actual accuracy.
Wire in Slack alerts so drift surfaces to your team immediately rather than requiring anyone to remember to check a dashboard.
Migrate to a managed platform (Arize, WhyLabs, Fiddler) once scale or explainability needs genuinely demand it — most teams start DIY with Evidently and move up only when they've actually outgrown it.
Common Mistakes People Make
Only monitoring feature drift and assuming that covers everything. Concept drift can occur with stable-looking inputs — watching only the easiest-to-detect layer leaves a genuine blind spot.
Generating reports nobody actually looks at. A drift report with no alerting attached is functionally equivalent to no monitoring — the alert, not the report, is what actually closes the loop.
Retraining automatically the instant any drift is detected, without diagnosing the cause. Seasonal shifts and upstream pipeline bugs both look like "drift" on paper — retraining blindly on either wastes effort or, worse, bakes a data bug directly into your model.
Treating drift monitoring as sufficient for regulatory compliance in a regulated industry. Statistical drift tools answer a different question than behavioral assurance — conflating the two leaves a real governance gap.
Running drift checks manually or "when someone remembers." This is exactly the churn-model story from this article's opening — automate the schedule, don't rely on human vigilance for something this important.
Recommended Books
- Designing Machine Learning Systems by Chip Huyen — the definitive guide to production ML monitoring, including drift detection patterns, alerting strategies, and the business-impact correlation that makes monitoring actionable.
- Machine Learning Engineering by Andriy Burkov — rigorous treatment of production ML lifecycle including monitoring, retraining triggers, and the distinction between statistical drift and genuine performance degradation.
- Practical MLOps by Noah Gomez — hands-on coverage of Evidently AI, monitoring pipelines, and integrating drift detection into CI/CD and continuous training workflows.
- Trustworthy Machine Learning — covers behavioral assurance and regulatory compliance for AI systems, the governance layer drift monitoring alone cannot address.
Wrapping This Up
Model monitoring closes the loop the rest of this MLOps arc opened: DVC versions your data, dbt and Airflow build and schedule your pipeline, CI/CD gates a new model before deployment — and monitoring is what tells you, continuously and automatically, whether the model that passed every one of those gates on day one is still doing its job on day two hundred. Data drift, concept drift, and prediction drift are genuinely distinct failure modes requiring layered detection, not a single dashboard number.
Remember that detecting drift is not the same as knowing what caused it, and that retraining should follow a genuine diagnosis rather than firing automatically the instant any statistical threshold trips. FYI, this genuinely closes the full production-ML loop this series has built piece by piece — from the first RAG tutorial's retrieval pipeline through Airflow's orchestration, dbt's transformations, Feast's feature consistency, DVC's versioning, CI/CD's quality gates, and now this article's continuous watchfulness after deployment — the complete lifecycle the very first MLOps article promised to explain, now actually built out end to end :)
Now go check whether any model you've deployed from earlier in this series has an actual reference dataset saved anywhere you could run a drift comparison against today. If the honest answer is no, that's genuinely the first, smallest step worth taking before anything else in this article — you can't detect drift against a baseline you never kept.