Train-Serve Skew - 5 min read

The Bug That Hides in Plain Sight

Your model scores 94% in eval. You ship it. Something feels off. You check the logs — predictions are coming out fine, no errors. But users are complaining.

The model is broken. And nothing is telling you that.

This is train-serve skew. And it's one of those bugs that's embarrassingly simple once you see it, but really easy to miss for way too long.

Here's the basic idea

When you train a model, your data goes through some preprocessing — normalization, feature engineering, encoding. The model learns from data shaped exactly like that.

When you deploy, you write another piece of code that prepares live inputs for inference.

These two are supposed to be identical. They never quite are.

That gap is the skew. The model has no way to tell you something is off — it just quietly gives you worse answers.

Let me make that concrete

Say you're building a recommendation model. One feature is avg_session_length.

• Training: computed from the user's full 2-year history in your data warehouse.
• Serving: computed from the last 7 days — because querying full history at runtime is too slow.

Same feature name. Completely different distribution. The model learned correlations assuming full history. Now it's getting 7-day windows it's never seen before.

Your offline metrics? Still 94%. Because they were computed using the training pipeline.

Nothing is broken visibly. That's the whole problem.

And it shows up in more ways than you'd expect

Feature skew — the values themselves differ (like above).

Pipeline skew — the logic diverges somewhere. You normalize with a fixed mean in training, but serving reimplements it with a running mean. Or training was Python and someone rewrote serving in Java and got something slightly wrong.

Schema skew — a field is a float in training but gets logged as int in production. A new category appears after deployment and your encoder maps it to unknown.

No exception thrown in any of these. Just silent drift.

The fix is almost philosophical

The root cause is that preprocessing lives in two places. So the fix is to make it live in one.

Scikit-learn Pipelines — serialize the whole pipeline (preprocessor + model) as one object. Same thing that trained is the same thing that serves.

TensorFlow Transform — bakes transformations into the model graph. The SavedModel includes the preprocessing. There is no second pipeline.

Feature stores — precompute features once, store them, use them at both training and serving time. One computation, two consumers.

The principle is simple: the model and its preprocessing should be one artifact — not two things staying in sync by convention.

Convention breaks. Make divergence impossible.