Feature Stores

"A feature, computed once offline, served a million times online. The day the two copies disagreed, the model started predicting a world that never existed."

A Feature Value That Time-Traveled by Accident

Every section of this chapter so far has treated features as if they simply arrived: the candidate generator of Section 38.3 read user and item embeddings, and the ranker of Section 38.4 consumed dozens of numerical signals per candidate. In a real recommender those numbers do not appear by magic. A user's "average dwell time over the last seven days", an item's "click-through rate this hour", a cross feature like "times this user bought from this category" are each the output of a distributed computation over event logs that can run to petabytes. The feature store is the system that owns those computations and their results. It exists because the alternative, letting the training pipeline and the serving pipeline each compute features their own way, produces two subtly different definitions of the same number and a model that fails silently in production.

1. One Definition, Two Stores Beginner

A feature store separates a feature's definition from its storage, and that separation is the whole idea. The definition is a piece of code, "count this user's clicks in the trailing hour", written once. The storage is split in two because training and serving have opposite access patterns. Training reads enormous slices of history: every example for every user over months, scanned sequentially, where throughput per byte matters and a query taking minutes is fine. Serving reads one row at a time under a hard deadline: given a user id, return that user's current feature vector in single-digit milliseconds, where latency per lookup is everything and a full scan is unthinkable. No single storage engine is good at both, so the feature store keeps two materializations of the same feature, an offline store and an online store, and the engineering discipline is ensuring they encode the identical definition.

The offline store holds the complete history of every feature value, typically as columnar files (Parquet on object storage, the layout of Chapter 8) partitioned by date so that a training job reads only the slice it needs. The online store holds only the latest value per entity key, in a low-latency key-value system such as Redis or Cassandra, where a lookup by user id or item id is a single hash probe. The batch and streaming pipelines write to both: the batch job backfills history into the offline store and refreshes the online store on a schedule, while the streaming job updates the online store continuously so that a click from one second ago is already reflected when the same user makes their next request. Figure 38.5.1 traces all four of these write paths.

2. Train/Serve Skew and Point-in-Time Correctness Intermediate

The failure a feature store is built to prevent has a name: train/serve skew, the gap between the feature value a model trains on and the value it is served at inference. Skew has two sources. The first, definitional skew, is two pieces of code computing the same feature differently; the single-definition discipline of Section 1 closes it. The second is subtler and is the heart of this section: temporal skew, where the training value reflects information that was not yet available when the prediction would have been made. This is label leakage, and it makes offline metrics look spectacular while production performance collapses, because the model has learned to lean on numbers it can never have in time.

The cure is point-in-time correctness. Each training label is an event with a timestamp $t$: a recommendation shown, a click that did or did not follow. The feature vector attached to that label must contain, for each feature, the value that was knowable strictly as-of $t$, never a value computed afterward. Formally, let a feature have a log of $(\tau_j, v_j)$ pairs, each a value $v_j$ that became valid at time $\tau_j$. The point-in-time correct value for a label at event time $t$ is

the most recent value whose validity time does not exceed $t$. This is exactly the as-of join, or backward $\texttt{merge\_asof}$, and it ties directly to the event-time semantics and watermarks of Chapter 9: the timestamp $\tau_j$ is the feature's event time, not the wall-clock time the pipeline happened to process it. A naive join that simply attaches each entity's latest feature value ignores $t$ entirely and sets $\phi(t) = v_{\text{last}}$, leaking every value with $\tau_j > t$ into the training row.

The code below makes the leak concrete on a six-row event log. It performs both joins, the naive "latest value" join and the point-in-time as-of join, against two labels stamped at 10:30, and reports the skew between what each join would train on and what the model could actually see at serving time.

import numpy as np
import pandas as pd

# Per user, a feature ("clicks in last hour") evolves over time. The value is
# only KNOWN as-of the time it was computed. A label is stamped at its own event
# time; attaching a value computed AFTER that time leaks the future.
feature_log = pd.DataFrame({
    "user": ["u1", "u1", "u1", "u2", "u2", "u2"],
    "feature_time": pd.to_datetime([
        "2026-01-01 09:00", "2026-01-01 10:00", "2026-01-01 11:00",
        "2026-01-01 09:00", "2026-01-01 10:00", "2026-01-01 11:00"]),
    "clicks_last_hour": [1, 2, 9, 0, 1, 8]})   # both spike late, after the label

labels = pd.DataFrame({
    "user": ["u1", "u2"],
    "event_time": pd.to_datetime(["2026-01-01 10:30", "2026-01-01 10:30"]),
    "bought": [1, 0]})

# NAIVE join: attach each user's LATEST feature, ignoring time. Both labels get
# the 11:00 value (9, 8), computed AFTER the 10:30 event. This is the leak.
latest = feature_log.sort_values("feature_time").groupby("user").tail(1)
naive = labels.merge(latest[["user", "clicks_last_hour"]], on="user", how="left")

# POINT-IN-TIME (as-of) join: most recent feature with feature_time <= event_time.
pit = pd.merge_asof(
    labels.sort_values("event_time"), feature_log.sort_values("feature_time"),
    by="user", left_on="event_time", right_on="feature_time", direction="backward")

# At serving the model only ever sees as-of values like the PIT column. Training
# on the NAIVE column teaches it from numbers it can never have in time.
print("NAIVE join (latest feature, leaks the post-event spike):")
print(naive.to_string(index=False))
print("\nPOINT-IN-TIME as-of join (feature_time <= event_time):")
print(pit[["user", "event_time", "bought", "feature_time", "clicks_last_hour"]].to_string(index=False))

print("\nfeature the model trains on vs the value available at serving (as-of 10:30):")
naive_seen = dict(zip(naive["user"], naive["clicks_last_hour"]))
pit_seen = dict(zip(pit["user"], pit["clicks_last_hour"]))
for u in ["u1", "u2"]:
    print(f"  {u}: NAIVE trains on {naive_seen[u]:>2}  |  PIT trains on {pit_seen[u]:>2}"
          f"  |  available at serving = {pit_seen[u]:>2}"
          f"   skew = {naive_seen[u] - pit_seen[u]:+d}")

NAIVE join (latest feature, leaks the post-event spike):
user          event_time  bought  clicks_last_hour
  u1 2026-01-01 10:30:00       1                 9
  u2 2026-01-01 10:30:00       0                 8

POINT-IN-TIME as-of join (feature_time <= event_time):
user          event_time  bought        feature_time  clicks_last_hour
  u1 2026-01-01 10:30:00       1 2026-01-01 10:00:00                 2
  u2 2026-01-01 10:30:00       0 2026-01-01 10:00:00                 1

feature the model trains on vs the value available at serving (as-of 10:30):
  u1: NAIVE trains on  9  |  PIT trains on  2  |  available at serving =  2   skew = +7
  u2: NAIVE trains on  8  |  PIT trains on  1  |  available at serving =  1   skew = +7

The skew of $+7$ in Output 38.5.1 is not a rounding artifact; it is the entire post-event spike leaking backward into the past. A model trained on the naive column would discover that "high clicks predicts a buy" and score beautifully offline, because in the training data the click spike and the buy co-occur. In production the spike has not happened yet at decision time, so the feature reads 2, not 9, and the learned relationship is worthless. This is why point-in-time correctness, not freshness, is the property that makes the recommender honest: a stale-but-as-of feature trains a model that works; a fresh-but-leaked feature trains a model that lies.

# Define once; Feast keeps offline and online materializations consistent.
from feast import FeatureStore
store = FeatureStore(repo_path=".")

# TRAINING: leakage-free join, the distributed version of Code 38.5.1.
training_df = store.get_historical_features(
    entity_df=labels,                      # columns: user, event_timestamp
    features=["user_stats:clicks_last_hour", "user_stats:dwell_7d"],
).to_df()                                  # each row's features are as-of its event_timestamp

# SERVING: same definitions, online store, millisecond lookup by key.
vec = store.get_online_features(
    features=["user_stats:clicks_last_hour", "user_stats:dwell_7d"],
    entity_rows=[{"user": "u1"}],
).to_dict()

3. Freshness, the Latency Budget, and the Pipelines That Fill the Store Intermediate

Point-in-time correctness governs the offline store and training; two other quantities govern the online store and serving. The first is the latency budget. The ranker of Section 38.4 sits inside a request that must return in tens of milliseconds, and every online feature lookup spends part of that budget. If the ranker scores $C$ candidates and the request allows $T_{\text{budget}}$ for feature retrieval, then with feature lookups of latency $\ell$ issued with batching factor $B$ (keys per round trip), the retrieval must satisfy

This inequality is why online feature stores live or die on tail latency and on batching: a $p99$ lookup of a few milliseconds, multiplied across hundreds of candidates, is the difference between a request that meets its deadline and one that times out. It is also why the online store is a key-value system and not a relational database; the access pattern is a batch of point lookups by entity key, exactly what Chapter 11 built sharded embedding tables to serve, and the same sharding-by-key logic places online feature rows across the cluster.

The second quantity is freshness, the lag between an event happening and its effect appearing in the online store. Define the freshness lag of a feature as

the time from when an event occurs to when the feature reflecting it can be read at serving. A feature populated only by a nightly batch job has a freshness lag of up to a day; a feature populated by a streaming job has a lag of seconds. Freshness trades off against cost and complexity: streaming pipelines are harder to operate than batch jobs and consume resources continuously, so a recommender computes slowly-changing features (a user's 30-day purchase history) in batch and fast-changing ones (clicks in the current session) in a streaming job. This batch-plus-streaming split is the classic two-pipeline pattern, and Section 38.6 takes up the streaming side in full, where freshness lag is the dominant design constraint.

The feature store closes the loop the chapter opened: candidate generation and ranking consume features, and the feature store is where those features are defined, computed by distributed batch and streaming pipelines, and served from a distributed low-latency store, with point-in-time correctness as the contract that lets the model trust its own training data. What remains is the fast edge of this picture, the streaming pipelines that drive freshness lag down to seconds so the recommender can react within a single session. That is the subject of Section 38.6.