Data Versioning and Lineage

"They asked me to reproduce last Tuesday's run. I had the code, the seed, and the config. I did not have last Tuesday's data, and so I had nothing at all."

A Worker That Trained on a Table That No Longer Exists

Every prior section of this chapter made the storage and loading path faster: object storage that scales past one disk (Section 8.2), columnar lakehouse tables (Section 8.3), partitioning and compaction (Section 8.4), sharded loaders (Section 8.5), streaming pipelines (Section 8.6), distributed preprocessing (Section 8.7), and leakage-safe correctness (Section 8.8). None of that speed is worth anything if you cannot say, six months later, exactly which bytes a deployed model was trained on. That question is not academic. It is asked by a regulator auditing a credit model, by an engineer bisecting a sudden accuracy regression, and by a researcher trying to reproduce a published number. The answer requires that the training dataset have a stable, verifiable identity, and that the chain from raw source to final table be recorded. Those two capabilities, versioning and lineage, are the subject of this section, and together they convert the fast-but-forgetful pipeline of the earlier sections into one you can trust.

1. A Model Is a Function of Its Data Beginner

Write a trained model as the output of a training procedure applied to three inputs: the code, the configuration (hyperparameters and seed), and the data. Symbolically, $\theta = \mathcal{T}(\text{code}, \text{config}, D)$, where $D$ is the exact training set. Reproducibility, the property defined and measured in Section 5.7, is the guarantee that re-running $\mathcal{T}$ with the same three inputs yields the same $\theta$ up to controlled nondeterminism. Practitioners habitually pin the first two inputs, a Git commit for the code and a config file for the seed and hyperparameters, and then leave the third, $D$, as a mutable path like s3://data/train/ whose contents drift from day to day. Pinning two of three inputs and letting the third float does not give you two-thirds of reproducibility; it gives you none, because $\theta$ depends on all three and a single relabeled row changes the result.

The fix is to give $D$ an identity as precise as a Git SHA. The dataset must be addressable not by a path that points at whatever currently lives there, but by a name that is bound to specific bytes and can never silently change underneath you. This is the same insight that made source control useful: a commit hash names a tree of bytes, not a branch that moves. Applied to data, it means the training table referenced by a run is a frozen snapshot, and the reference is to that snapshot's fingerprint, so "the data this model saw" is a question with one unambiguous answer forever.

2. Content-Addressed Snapshots and Dataset Fingerprints Intermediate

The mechanism that gives data a stable identity is content addressing: name a blob by a cryptographic hash of its bytes rather than by a location. A snapshot of a dataset is the ordered collection of its records, and its fingerprint is a hash computed over the canonical bytes of every record in order. Two properties follow immediately and are exactly the two we need. First, identity is reproducible: rebuilding the same logical dataset from scratch, even on a different machine, yields the same fingerprint, because the hash sees only content. Second, identity is sensitive: changing a single label, dropping one row, or reordering records changes the fingerprint completely, so a snapshot's name is a tamper-evident witness to its exact contents. This is the same Merkle-style addressing that underpins Git and content-addressable object stores; here we point it at training data.

The program below builds three snapshots of a tiny labeled dataset and fingerprints each. Snapshot v1 is the baseline. Snapshot v2 has identical content rebuilt independently (with record keys deliberately reordered, to show the fingerprint depends on content, not on incidental serialization order). Snapshot v3 flips one single label. It then writes a lineage record binding the inputs and the transform code to the output snapshot, the topic of the next section.

import hashlib
import json

# A dataset snapshot is an ordered set of records. Its IDENTITY is a content hash:
# the same bytes always produce the same fingerprint, any change produces a new one.

def fingerprint(records):
    """Content-address a dataset snapshot: hash the canonical bytes of every record."""
    h = hashlib.sha256()
    for r in records:                                  # order-stable iteration
        canonical = json.dumps(r, sort_keys=True, separators=(",", ":"))
        h.update(canonical.encode("utf-8"))
        h.update(b"\n")                                # record delimiter
    return h.hexdigest()

# Snapshot v1: three labeled training rows.
v1 = [
    {"id": 1, "text": "ship the model", "label": "pos"},
    {"id": 2, "text": "the run crashed", "label": "neg"},
    {"id": 3, "text": "all-reduce converged", "label": "pos"},
]

# Snapshot v2: identical content, rebuilt independently (reproducible identity).
v2 = [
    {"label": "pos", "text": "ship the model", "id": 1},   # keys reordered on purpose
    {"id": 2, "text": "the run crashed", "label": "neg"},
    {"id": 3, "text": "all-reduce converged", "label": "pos"},
]

# Snapshot v3: ONE label flipped on row 2 (a single-cell change).
v3 = [dict(r) for r in v1]
v3[1]["label"] = "pos"

fp1, fp2, fp3 = fingerprint(v1), fingerprint(v2), fingerprint(v3)
print("snapshot v1 fingerprint :", fp1[:16], "...")
print("snapshot v2 fingerprint :", fp2[:16], "...")
print("snapshot v3 fingerprint :", fp3[:16], "...")
print("v1 == v2 (same content) :", fp1 == fp2)
print("v1 == v3 (one cell flip):", fp1 == fp3)

# A lineage record links INPUTS + CODE -> the OUTPUT snapshot it produced.
raw_sources = ["s3://corpus/reviews-2026-06.jsonl", "s3://corpus/labels-v4.csv"]
code_version = "git:9f3a1c2"                            # the transform that built it
lineage = {
    "output_snapshot": fp1,
    "inputs": raw_sources,
    "transform_code": code_version,
    "row_count": len(v1),
}
# The lineage record is itself content-addressed, so the (inputs, code) -> output
# binding is tamper-evident: change any field and its own id changes too.
lineage_id = hashlib.sha256(
    json.dumps(lineage, sort_keys=True).encode("utf-8")
).hexdigest()[:16]
print("lineage record id       :", lineage_id)
print("lineage record          :", json.dumps(lineage, indent=None))

snapshot v1 fingerprint : 3dd4f59af2233d0a ...
snapshot v2 fingerprint : 3dd4f59af2233d0a ...
snapshot v3 fingerprint : 38112a84caf59356 ...
v1 == v2 (same content) : True
v1 == v3 (one cell flip): False
lineage record id       : 8daa463f2483324b
lineage record          : {"output_snapshot": "3dd4f59af2233d0a45a0b32bc487a1b7badc99f74221c1ecda853e0cb6b1712e", "inputs": ["s3://corpus/reviews-2026-06.jsonl", "s3://corpus/labels-v4.csv"], "transform_code": "git:9f3a1c2", "row_count": 3}

The two outcomes in Output 8.9.1 are the whole argument. The same content gives the same name no matter where or when it was built, so a fingerprint is a stable handle you can record in a run's metadata and resolve forever. A one-cell change gives a different name, so the fingerprint cannot be fooled: if the data a model saw differs in any record from the data you are holding now, the mismatch is visible at a glance. A real implementation hashes file chunks or table manifests rather than in-memory dicts, and stores the bytes once under their hash in an object store, but the identity guarantee is precisely the one this toy demonstrates.

3. Snapshots in Practice: Time Travel, DVC, and lakeFS Intermediate

Hashing every record by hand, as in Code 8.9.1, explains the idea but does not scale to a multi-terabyte table that thousands of workers read concurrently. Three production mechanisms give you the same immutable-snapshot guarantee at scale, and they differ mainly in what granularity they version. The first is lakehouse time travel. The Delta Lake and Apache Iceberg table formats from Section 8.3 keep a transaction log of immutable snapshots; every commit produces a new table version, and a reader can ask for the table as of a specific version or timestamp. A training job records the table version it read, and that version is a frozen snapshot the format guarantees will never change, because new writes append new snapshots rather than mutating old ones. The second is file-level versioning with DVC, which stores large files in an object store keyed by content hash and keeps small pointer files in Git, so a dataset version is just a Git commit that resolves to exact bytes. The third is branch-and-commit versioning of an entire object-storage bucket with lakeFS, which gives a whole data lake Git-like semantics (branch, commit, merge, revert) so you can snapshot, branch, and roll back petabytes the way you branch code. Table 8.9.1 places the three against the from-scratch fingerprint so you can match a mechanism to a granularity.

All four entries in Table 8.9.1 deliver the identity guarantee of Section 2; they differ in granularity and operational fit, not in the underlying idea. Lakehouse time travel is the right reach when the data already lives in Delta or Iceberg tables, because the snapshot is a free byproduct of the transaction log you are already keeping. DVC fits file-based corpora that you want to version in the same pull request as the code that consumes them. lakeFS fits an organization that wants Git-like branching over an entire lake, so that a risky relabeling can happen on a branch and be merged only once a model trained on it has been validated.

# 1) Lakehouse time travel: read a table AS OF an exact version (deltalake / Delta).
from deltalake import DeltaTable
dt = DeltaTable("s3://lake/train_table")
version = dt.version()                         # the immutable snapshot this run reads
df = dt.to_pyarrow_table()                     # ... reproducible: pin `version` in metadata
# later, reproduce exactly:  DeltaTable("s3://lake/train_table", version=version)

# 2) DVC: snapshot a file dataset; its identity is the resulting Git commit.
#   $ dvc add data/train.parquet      # content-hashes the file, writes a .dvc pointer
#   $ git commit -am "train snapshot" # the commit IS the dataset version

# 3) lakeFS: commit the whole bucket on a branch; the commit id names the snapshot.
import lakefs
branch = lakefs.repository("corpus").branch("main")
commit = branch.commit(message="frozen train snapshot")   # commit.get_commit().id

4. Lineage: From Raw Source to Trained Model Intermediate

A snapshot tells you what the data was; lineage tells you where it came from. Lineage is the recorded chain linking each artifact to the inputs and the code that produced it: which raw sources, which transformations, which commit of which job built the training table that produced a given model. The lineage record at the bottom of Code 8.9.1 is the atomic unit, one output snapshot bound to its inputs and its transform code, and a real pipeline records one such edge per transformation, so the edges compose into a graph from raw sources all the way to the deployed model. This is the same idea you met as RDD lineage in Section 7.2, where Spark records how each partition was derived from its parents so a lost partition can be recomputed. There, lineage served fault tolerance (recompute what was lost). Here, the same recorded provenance serves reproducibility and audit: trace a data issue to its origin, and reproduce or roll back from there.

The payoff is concrete. When a model's accuracy drops on one customer segment, lineage lets you walk backward from the model to the training snapshot, from the snapshot to the transform that built it, and from the transform to the specific raw source, perhaps a relabeled file or a corrupted upstream feed, that introduced the defect. Without lineage you are reduced to guessing; with it, debugging a data problem becomes a graph traversal. And because every node in that graph is a content-addressed snapshot, you can not only find the bad input but reproduce the exact state of the world before it entered, retrain from the last good snapshot, and prove the fix. Figure 8.9.1 draws the graph this section has been building toward.

5. From Data Lineage to Experiment Tracking Intermediate

The lineage record of Code 8.9.1 stops at the dataset snapshot, but the natural next edge in the graph runs from the snapshot to the model and from the model to the experiment that produced it. That extension is the province of MLOps experiment tracking, where each run logs its code commit, its config, its dataset snapshot id, and the resulting metrics and model artifact, so the entire chain from raw source to deployed model is queryable. The dataset fingerprint this section produces is exactly the field an experiment tracker needs in order to answer "which data did this production model see?", and pinning it is what makes the tracker's reproducibility claim true rather than aspirational. We develop that full model-and-experiment provenance, including model registries and the audit trails a deployed fleet must keep, in Chapter 26. For now the boundary is clean: this chapter versions the data and records the edges up to the training set; MLOps versions the model and the experiment and records the edges beyond it, and the dataset fingerprint is the handshake between the two.

6. Chapter Summary and Project Ideas Beginner

This section closes Chapter 8, so it is worth tracing the spine the whole chapter built. The storage layer, not the training loop, is what determines whether an AI system scales (Section 8.1), because the bytes have to be read before any gradient can be computed. We built that layer from the bottom up: object storage and distributed filesystems that scale capacity past one disk (Section 8.2); columnar formats and the lakehouse, Parquet and Arrow under Delta and Iceberg, that make analytical reads and atomic table snapshots cheap (Section 8.3); data layout, partitioning, and compaction that decide how much of a table a query must touch (Section 8.4); sharded training data and the loader that keeps accelerators fed rather than starved (Section 8.5); streaming and WebDataset-style pipelines that read shards as a sequential stream (Section 8.6); distributed preprocessing for the transformations that must themselves scale out (Section 8.7); the leakage and correctness traps that a distributed pipeline introduces and must guard against (Section 8.8); and finally, in this section, the versioning and lineage that make every layer below reproducible and auditable. The thread tying them together is that storage is not a passive substrate but an active part of the distributed AI system, and that the same data which determines a model's behavior must therefore be sharded for throughput, kept correct under concurrency, and named precisely enough to reproduce.

That closes Chapter 8, and with it the storage and data-loading foundation of Part II. You arrived able to load a file; you leave able to build a distributed data path that scales in capacity, reads cheaply, feeds accelerators without starving them, stays correct under concurrency, and names its every snapshot precisely enough to reproduce and audit. The data so far has been at rest: bounded tables, read in full, versioned as static snapshots. The next chapter sets the data in motion. Chapter 9 asks what changes when the dataset is unbounded and arriving continuously, when "the training set" is a never-ending feed of events, and when a model must update and serve in real time rather than from a frozen snapshot. The storage discipline of this chapter becomes the durable log beneath that stream, and the versioning instinct you built here becomes the watermark and checkpoint machinery that keeps a streaming computation correct.