Executive Summary

A machine learning model can achieve excellent accuracy during development and still underperform immediately after deployment.

One of the most common reasons is training–serving skew: the features presented to the model during training are different from the features it receives in production.

The model itself may be functioning correctly. The APIs may be healthy. Infrastructure monitoring may show no failures. Yet the predictions can gradually become less reliable because the model is operating on inputs that differ from the data distribution it learned from.

Training–serving skew commonly results from differences in preprocessing, feature engineering, data sources, timestamps, missing-value handling, or feature freshness. Google identifies these differences as a major source of production ML failures and recommends continuously comparing training and serving data rather than assuming that the two pipelines remain consistent.

The enterprise solution is to treat ML features as governed, versioned production assets.

An enterprise feature-store architecture provides a common layer for defining and managing features while supporting two different operational requirements:

• Offline storage for historical, point-in-time-correct training data

• Online storage for low-latency production inference

The objective is straightforward:

Create the feature once, set clear rules for it, and use it the same way throughout the ML process.

The Training–Serving Skew Problem

Consider a model that predicts whether a customer is likely to make a purchase.

During training, the model receives:

customer_spend_30d = SUM(transactions over previous 30 days)

The production service independently implements the same feature:

customer_spend_30d = SUM(transactions over previous 28 days)

The model does not know that the implementation changed.

It simply receives a different input.

The problem can therefore look like this:

This is particularly dangerous because nothing necessarily fails technically.

The API still returns 200 OK.

The model still produces predictions.

The infrastructure may still report healthy CPU, memory and network utilization.

The failure exists in the meaning of the data being supplied to the model.

Google defines training–serving skew as a difference between model performance during training and serving and identifies discrepancies between training and serving pipelines as a key cause.

Where the Skew Comes From?

The most common causes are relatively simple:

1. Different transformation logic

   Training may use a Python or SQL transformation while the production API implements the same feature independently.

2. Different missing-value handling

   Training: NULL → median

   Production:NULL → 0

3. Different time semantics

   Training may calculate a rolling feature using UTC event timestamps while serving uses local timestamps or processing time.

4. Stale reference data

   Training may use an updated lookup table while production continues reading an older version.

5. Different feature availability

   A feature may exist reliably in historical training data but be unavailable during a portion of live inference requests.

These differences are often introduced gradually as individual systems evolve. The result is not necessarily an application failure; it is a silent change in what the model sees.

Why Independent Feature Pipelines Fail

A common architecture looks like this:

This design creates two sources of truth.

Initially, Feature Logic A and Feature Logic B may be identical.

Over time:

• one transformation changes;

• one dependency is upgraded;

• one team changes a default;

• one pipeline receives a new data source;

• one system handles missing values differently.

The two implementations then diverge.

Google's machine-learning guidance recommends avoiding this pattern by ensuring that training uses the same feature representation that is available during serving and by explicitly monitoring the difference.

This is where an enterprise feature store becomes valuable.

Enterprise Feature Store Architecture

A feature store provides a governed layer between raw enterprise data and ML models.

The architecture separates historical feature retrieval from real-time feature serving, while keeping feature definitions and metadata under centralized control.

Reference Architecture

The important architectural decision is that training and serving should not independently reinvent feature logic.

The offline and online stores serve different purposes, but the feature definition should remain governed as one logical asset.

Snowflake's Feature Store, for example, centralizes feature transformations and supports both batch and streaming data, point-in-time-correct feature generation and ML lineage.

Offline and Online Feature Stores

The two-store pattern exists because training and inference have fundamentally different requirements.

Offline Store

The offline store is optimized for:

• Historical data

• Large-scale analytical queries

• Training dataset generation

• Backfills

• Point-in-time joins

Typical technologies include:

BigQuery or Snowflake.

The training pipeline might query:

The critical requirement is that the training dataset must reproduce the features that would actually have been available at the time of the prediction.

Online Store

The online store is optimized for:

• Low-latency lookups

• High request volume

• Current feature values

• Real-time inference

Redis is a practical choice for this layer because it is designed for low-latency online feature retrieval and supports batch and streaming feature updates. Redis documents sub-millisecond feature serving as a target for production online feature-store workloads.

The inference path becomes:

The production application therefore does not need to repeatedly reconstruct complex features from operational databases.

Unified Feature Contracts

The feature itself should be treated as an API contract.

Instead of documenting only customer_spend_30d an enterprise definition should specify:

This contract establishes exactly what the feature means.

It also creates a boundary for change.

If the calculation changes from SUM(transaction_amount) to SUM(transaction_amount after currency normalization) that should be treated as a feature-version change, not an invisible implementation detail.

For example:

customer_spend_30d:v1
        │
        ├── Model A
        └── Model B

customer_spend_30d:v2
        │
        ├── Model C
        └── Model D

Feature-store platforms such as Feast explicitly support feature definitions and feature-service versioning, while Snowflake provides feature views and ML lineage for managing feature dependencies.

Point-in-Time Correctness

Feature consistency is not only about calculating the same formula.

It is also about calculating it using only information that was available at the time of prediction.

Consider a prediction made at 10:00 AM.

The training pipeline must not accidentally retrieve a feature generated at 11:00 AM because that value contains information that would not have existed when the model was actually making the decision.

Point-in-time joins ensure that historical training examples use the feature value that was available at the relevant timestamp.

Feast explicitly supports point-in-time-correct joins to reproduce the state of features at a specific point in the past and prevent future values from leaking into training data.

Snowflake similarly supports point-in-time-correct feature generation using time-aware joins.

This makes point-in-time correctness both:

• a training integrity control, and

• a data-leakage prevention mechanism.

Detecting Offline–Online Inconsistency

A feature store reduces the opportunity for skew, but enterprises should still measure feature consistency.

A practical validation flow is:

For deterministic numerical features:

abs(offline_value - online_value) < tolerance

For categorical features:

offline_category == online_category

Enterprises should also monitor:

• Feature freshness

• Missing-value rate

• Feature coverage

• Distribution differences

• Unexpected schema changes

Google's production ML guidance recommends explicitly monitoring training and serving data because skew can remain hidden while the system continues generating predictions.

Enterprise Implementation Blueprint

A practical implementation can be reduced to six steps.

1. Inventory Existing Features

Identify:

• Training features

• Production features

• Transformation code

• Source systems

• Models consuming each feature

Create a dependency chain:

2. Centralize Feature Definitions

Move duplicated feature calculations into a governed feature layer.

The goal is:

rather than:

3. Establish the Offline Store

Use a warehouse such as BigQuery or Snowflake for:

• Historical features

• Training datasets

• Backfills

• Point-in-time retrieval

4. Establish the Online Store

Use Redis or another low-latency online store for:

• Current feature values

• Real-time inference

• High-volume feature retrieval

Redis specifically documents the online feature-store pattern of serving precomputed features under tight latency requirements while receiving both batch and streaming updates.

5. Enforce Feature Contracts

Every production feature should have:

• Schema

• Owner

• Version

• Transformation definition

• Freshness requirement

• Null policy

Breaking changes should fail validation before deployment.

6. Continuously Validate

Compare the feature values and distributions used during training with those observed during production inference.

The objective is to detect skew before it becomes a model-performance incident.

Real-World Evidence: Google Play

One of the clearest demonstrations of the business impact of training–serving skew comes from Google Play.

Google's production ML platform compared statistics from serving logs with the corresponding training data and discovered features that were consistently present during training but missing during serving.

An online A/B experiment showed that removing the skew improved the app-install rate on the main Google Play landing page by 2%.

The significance is larger than the percentage itself.

The model was not necessarily broken.

The serving infrastructure was not necessarily broken.

The problem was that the model was receiving a different set of inputs than the training process had assumed.

This is precisely why feature consistency needs to be treated as a production reliability concern, not simply a data-science concern.

Enterprise Best Practices

A production feature-store architecture should enforce a small set of non-negotiable controls:

Define features once

Avoid independently implementing the same transformation in training and serving.

Version feature definitions

A change in feature semantics should create a new version rather than silently modifying an existing production feature.

Enforce point-in-time correctness

Historical training data must contain only information available at prediction time.

Monitor freshness

A mathematically correct feature can still be operationally wrong if it is several hours old.

Validate offline/online parity

Regularly compare feature values and distributions between training and production.

Maintain ownership and lineage

Every production feature should have an accountable owner and a traceable path from source data to model.

These controls align with the broader feature-store objective described by Snowflake: centralizing feature transformations, improving consistency and freshness, and maintaining lineage across source data, features, datasets and models.

The Enterprise Architecture Principle

The key architectural shift is simple:

becomes:

The physical storage can be different.

The feature contract cannot be.

That distinction is fundamental.

BigQuery or Snowflake may provide the historical representation, while Redis provides the low-latency representation. The enterprise feature platform ensures that both are governed around the same feature definitions and semantics.

Conclusion

Training–serving skew is one of the most dangerous ML production failures precisely because it is often invisible.

The model continues running.

The APIs continue responding.

The infrastructure continues reporting healthy.

But the model is making decisions using data that does not match what it learned during training.

An enterprise feature store addresses this problem by establishing a governed feature layer across the complete ML lifecycle:

• Centralized feature definitions

• Offline historical storage

• Low-latency online serving

• Point-in-time correctness

• Versioned feature contracts

• Feature freshness controls

• Offline/online consistency validation

• Lineage and ownership

The objective is not simply to store features.

It is to ensure that the feature used to train the model and the feature used to make a production decision represent the same thing.

For enterprise ML systems, that consistency is not an optimization.

It is a reliability requirement.

References

Google for Developers. (2024). Rules of machine learning: Best practices for ML engineering. Google. https://developers.google.com/machine-learning/guides/rules-of-ml/

Google Cloud. (2021). Monitor models for training-serving skew with Vertex AI. Google Cloud. https://cloud.google.com/blog/topics/developers-practitioners/monitor-models-training-serving-skew-vertex-ai

Breck, E., Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2019). Data validation for machine learning. Proceedings of Machine Learning and Systems. Google Research. https://research.google/pubs/data-validation-for-machine-learning/

Feast. (2026). Point-in-time joins. Feast Documentation. https://docs.feast.dev/master/getting-started/concepts/point-in-time-joins

Feast. (2026). Build a training dataset. Feast Documentation. https://docs.feast.dev/master/how-to-guides/feast-snowflake-gcp-aws/build-a-training-dataset

Redis. (2026). Redis as a feature store. Redis Documentation. https://redis.io/docs/latest/develop/use-cases/feature-store/

Snowflake. (2026). Feature store. Snowflake Documentation. https://docs.snowflake.com/en/developer-guide/snowflake-ml/feature-store/

Snowflake. (2026). Working with feature views. Snowflake Documentation. https://docs.snowflake.com/en/developer-guide/snowflake-ml/feature-store/feature-views

Google Cloud. (2023). How Wayfair improves its feature engineering with Vertex AI. https://cloud.google.com/blog/products/ai-machine-learning/how-wayfair-improves-its-feature-engineering-with-vertex-ai