Vector Databases

"A raw index can tell you which vectors are nearest. A vector database remembers them tomorrow, lets you delete the embarrassing ones, and never serves a stale neighbor without admitting it."

An Index Shard That Grew Up Into a Database

The previous section built the distributed embedding pipeline that turns a corpus into a stream of vectors. Those vectors have to live somewhere that can be searched at serving time, and "somewhere" is the subject here. It is tempting to reach for a bare nearest-neighbor library, load the vectors into memory, and call it done. That works until the first hard question arrives from production: where do the vectors go when the process restarts, what happens when a document is edited or deleted, how do you search only the items a particular user is allowed to see, and what do you do when the collection outgrows one machine. Each of those questions is the difference between an index and a database, and answering all of them at once across a cluster is what dedicated vector databases exist to do.

We approach the vector database as a distributed system first and a search engine second. The nearest-neighbor algorithms themselves, the graph and quantization indexes that make a single shard fast, are deepened in Section 25.4; here we treat one shard's index as a black box that answers local queries and ask how to wrap many such boxes into a coherent, durable, filterable service. That framing keeps the distributed-systems content in front: storage, consistency, sharding, replication, and the coordinator that ties them together.

1. What a Vector Database Adds Over a Raw Index Beginner

A raw approximate-nearest-neighbor (ANN) index, of the kind introduced for classical machine learning in Section 12.7, solves exactly one problem: given a query vector, return the $k$ stored vectors most similar to it, quickly and approximately. That is necessary but far from sufficient for a production retrieval system. Five capabilities separate a vector database from the index it contains, and each one is a distributed-systems problem rather than a search problem.

The first is durable storage. An in-memory index vanishes on restart; a database persists the vectors and the index structure to disk or object storage, and can reload or rebuild after a crash without re-embedding the whole corpus. The second is CRUD: create, read, update, and delete. New documents arrive continuously, old ones get edited, and some must be removed outright, for correctness or for compliance. Inserting into and especially deleting from a graph-based ANN index is genuinely hard, because the graph's carefully built neighbor links assume a fixed point set; we return to this tension in Section 6. The third is metadata filtering: searching not the whole collection but the subset matching a predicate, such as documents owned by one tenant, written after a date, or tagged with a language. The fourth is horizontal scaling through sharding and replication, so the collection can exceed one machine's memory and the query rate can exceed one machine's throughput; this is the focus of Section 25.5. The fifth is consistency and freshness: a guarantee about how soon a newly inserted vector becomes searchable, and whether two replicas can disagree.

2. The Architecture: Coordinator, Shards, Replicas Beginner

A distributed vector database has the same shape as most distributed data stores. The collection of vectors is partitioned into shards, each shard holding a slice of the data and its own local ANN index over only that slice. Each shard is replicated onto several machines so that queries can be served from any replica and the loss of one machine does not lose the shard; replication for availability and read throughput is the same idea developed for distributed systems in Section 2.3. A query coordinator sits in front: it receives a query, scatters it to one replica of every shard, waits for each shard's local top-$k$, and merges those partial results into the global top-$k$ it returns to the client. A separate write path routes each new or changed vector to its owning shard and updates that shard's index, propagating the change to the shard's replicas.

The query side is a scatter-gather, and the merge is a small but exact piece of reasoning. If every shard returns its own top-$k$ by similarity, the true global top-$k$ is guaranteed to be a subset of the $S \times k$ candidates returned, because any vector in the global top-$k$ must also be in the top-$k$ of whichever shard happens to hold it. So the coordinator can merge correctly by sorting the union of the partial results and keeping the best $k$. Writing $s(q, v)$ for the similarity between query $q$ and vector $v$, and $\mathcal{S}_i$ for the set of vectors on shard $i$, each shard returns

which equals the single-machine $\operatorname{top\text{-}k}_{v} s(q, v)$ exactly when each shard's local search is exact. With approximate per-shard search the same merge applies; the approximation lives entirely inside each shard, not in the combine step. Figure 25.3.1 shows the full path.

3. A Sharded Vector Store From Scratch Intermediate

To make the architecture concrete, the code below builds a minimal sharded vector store entirely in pure Python and NumPy. Each shard holds a contiguous slice of the collection, its vectors, and a per-vector metadata tag. A shard's local search is an exact brute-force scan, standing in for the ANN index that Section 25.4 develops; the point here is the distributed wrapper, not the index. The coordinator scatters the query to every shard, gathers each shard's local top-$k$, and merges. We time the query two ways, with shards searched sequentially on one core and with shards searched concurrently so that each shard's scan overlaps the others, the difference that real fan-out across machines buys you.

import time
from concurrent.futures import ThreadPoolExecutor
import numpy as np

rng = np.random.default_rng(7)
N, D, K = 200_000, 128, 10          # vectors, dimension, top-k returned

# A synthetic collection: unit-normalised vectors plus a categorical "lang" tag.
data = rng.standard_normal((N, D)).astype(np.float32)
data /= np.linalg.norm(data, axis=1, keepdims=True)
langs = rng.integers(0, 5, size=N)   # metadata: 5 language buckets, 0..4

class Shard:
    """One shard: a contiguous slice of the collection it alone can search."""
    def __init__(self, ids, vecs, meta):
        self.ids, self.vecs, self.meta = ids, vecs, meta

    def search(self, q, k, lang=None):
        v, ids, meta = self.vecs, self.ids, self.meta
        if lang is not None:                 # metadata filter: search a subset
            mask = meta == lang
            v, ids = v[mask], ids[mask]
        if len(ids) == 0:
            return np.empty(0), np.empty(0, dtype=np.int64)
        sims = v @ q                          # cosine similarity (q is unit norm)
        top = np.argpartition(-sims, min(k, len(sims) - 1))[:k]
        return sims[top], ids[top]

def build_shards(s):
    """Partition the collection into s shards by contiguous id ranges."""
    bounds = np.array_split(np.arange(N), s)
    return [Shard(b, data[b], langs[b]) for b in bounds]

_POOL = ThreadPoolExecutor(max_workers=20)

def coordinator_search(shards, q, k, lang=None, parallel=False):
    """Scatter q to every shard, gather local top-k, merge into a global top-k."""
    if parallel:                              # shards run concurrently (one per worker)
        results = list(_POOL.map(lambda sh: sh.search(q, k, lang), shards))
    else:                                     # sequential fan-out on one core
        results = [sh.search(q, k, lang) for sh in shards]
    sims = np.concatenate([r[0] for r in results])    # gather
    ids = np.concatenate([r[1] for r in results])
    order = np.argsort(-sims)[:k]             # merge: global top-k by similarity
    return ids[order], sims[order]

The driver below builds the store at several shard counts, times sequential versus concurrent fan-out, measures the cost of a metadata-filtered query against an unfiltered one, and checks that the merged top-$k$ equals a single-machine brute-force top-$k$.

q = rng.standard_normal(D).astype(np.float32)
q /= np.linalg.norm(q)
REPS = 50

def timed(shards, k, lang=None, parallel=False):
    coordinator_search(shards, q, k, lang, parallel)    # warm up
    t0 = time.perf_counter()
    for _ in range(REPS):
        coordinator_search(shards, q, k, lang, parallel)
    return (time.perf_counter() - t0) / REPS * 1e3      # ms/query

print(f"collection: N={N:,} vectors, D={D}, top-k={K}\n")
print("query latency vs shard count (unfiltered):")
print(f"{'shards':>7} {'vecs/shard':>12} {'seq ms':>9} {'parallel ms':>13}")
for s in (1, 2, 5, 10, 20):
    shards = build_shards(s)
    print(f"{s:>7} {N // s:>12,} {timed(shards, K):>9.2f} "
          f"{timed(shards, K, parallel=True):>13.2f}")

shards = build_shards(10)
unf, flt = timed(shards, K), timed(shards, K, lang=2)
sel = float((langs == 2).mean())
print("\ncost of filtered search (10 shards):")
print(f"  unfiltered           : {unf:6.2f} ms/query (scans all {N:,})")
print(f"  filtered lang==2      : {flt:6.2f} ms/query (selectivity {sel:.2f})")

sims = data @ q                                          # single-machine ground truth
brute = set(np.argsort(-sims)[:K].tolist())
ids, _ = coordinator_search(build_shards(10), q, K)
print(f"\nmerged top-{K} matches single-machine top-{K}: {set(ids.tolist()) == brute}")

collection: N=200,000 vectors, D=128, top-k=10

query latency vs shard count (unfiltered):
 shards   vecs/shard    seq ms   parallel ms
      1      200,000     42.25         34.42
      2      100,000     35.56         21.67
      5       40,000     36.97         10.44
     10       20,000     37.91          8.92
     20       10,000     37.94          6.99

cost of filtered search (10 shards):
  unfiltered           :  36.33 ms/query (scans all 200,000)
  filtered lang==2      :  22.21 ms/query (selectivity 0.20)

merged top-10 matches single-machine top-10: True

Three lessons fall out of Output 25.3.1, and all three carry over to real vector databases. First, sharding alone does not lower single-query latency; the sequential column barely moves, because partitioning the same vectors into more groups does not change how many similarities must be computed. The latency win comes from running the shards in parallel, which here is threads over a NumPy matmul that releases the interpreter lock, and in production is independent machines. Second, metadata filtering is cheaper, not more expensive, when the filter is selective and applied before the distance computation, because each shard scans a smaller candidate set; the subtlety, taken up in Section 6, is that a graph ANN index cannot simply skip filtered-out nodes the way a brute-force scan can. Third, the merge is exact: scatter-gather over shards returns precisely what one big machine would have returned, exactly as the equation in Section 2 promised.

4. Metadata Filtering: Search Within a Subset Intermediate

Pure similarity is rarely the whole query. Real retrieval asks for the nearest vectors among the documents this user may see, or written in the last week, or belonging to this product category. Combining a similarity search with a metadata predicate is where vector databases earn much of their keep, and there are two honest strategies, each with a failure mode. Pre-filtering restricts the candidate set to the rows matching the predicate and then searches only those, which is what Code 25.3.1 does; it is exact and gets cheaper as the filter tightens, but it defeats a graph ANN index, whose neighbor links may all point at filtered-out nodes, forcing a fallback to brute force on the surviving subset. Post-filtering runs the ANN search first and then discards results that fail the predicate, which keeps the fast index but can return fewer than $k$ results, or none, when the predicate is selective and the nearest neighbors happen not to satisfy it.

Production systems navigate between these with techniques such as filtered HNSW traversal, per-filter sub-indexes for common predicates, and over-fetching (retrieve $c \cdot k$ then post-filter, picking $c$ from the filter's selectivity). The right choice depends on selectivity, the fraction of the collection that passes the filter, and on whether that selectivity is known in advance. A filter that keeps 80 percent of the data is best handled by post-filtering; a filter that keeps 0.1 percent is best handled by pre-filtering or a dedicated sub-index. The lesson from Output 25.3.1 holds either way: filtering changes how much work each shard does, so its cost must be modeled per shard, not as an afterthought bolted onto a finished search.

5. The Landscape: Libraries, Databases, and Extensions Beginner

The tools that store and serve vectors fall into three families, and choosing among them is the central build-versus-buy decision of a retrieval system. Libraries such as FAISS give you a fast in-process ANN index and nothing else: no durability, no CRUD, no metadata filtering, no network service, no sharding. They are ideal when the collection fits in memory, changes rarely, and lives inside a single application that already owns its persistence and scaling. Dedicated vector databases such as Milvus, Qdrant, Weaviate, and Pinecone wrap an index in exactly the distributed-systems machinery this section describes: durable storage, CRUD, metadata filtering, sharding, replication, and a query coordinator, exposed as a network service. They are the right tool when the collection is large, mutates continuously, must be filtered, and has to scale and stay available. Vector extensions to existing databases, such as pgvector for PostgreSQL or the dense-vector fields in Elasticsearch and OpenSearch, add similarity search to a system you already run, inheriting its transactions, backups, replication, and operational familiarity at the cost of ANN features that lag the dedicated systems.

The decision is rarely about raw search speed. It is about where the surrounding properties come from. If you already operate PostgreSQL and your vector collection is modest and naturally joins your relational data, pgvector means one fewer system to run and one consistent backup story. If vectors are your primary workload at scale, a dedicated vector database gives you sharding and filtered search that a general-purpose database bolts on awkwardly. If you are inside a tight loop with a fixed, in-memory index, a library skips the network entirely. Table 25.3.1 lays out the trade-off.

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
client.create_collection(                       # sharding, replication, persistence: configured, not coded
    "docs",
    vectors_config=models.VectorParams(size=128, distance=models.Distance.COSINE),
    shard_number=10, replication_factor=2,
)
client.upsert("docs", points=[                  # CRUD: insert/update by id, delete is one call
    models.PointStruct(id=i, vector=vec.tolist(), payload={"lang": int(lang)})
    for i, (vec, lang) in enumerate(zip(data, langs))
])
hits = client.query_points(                     # scatter-gather + filtered top-k: one call
    "docs", query=q.tolist(), limit=10,
    query_filter=models.Filter(must=[models.FieldCondition(
        key="lang", match=models.MatchValue(value=2))]),
).points

6. Freshness Versus Index Quality Advanced

The hardest tension in a vector database is between accepting new vectors quickly and keeping the index high quality. A graph-based ANN index such as HNSW (detailed in Section 25.4) is built by carefully wiring each vector to good neighbors, and that wiring assumes a roughly fixed point set. Inserting a vector means finding its neighbors and patching the graph, which is feasible but degrades graph quality over many insertions; deleting a vector is worse, because the graph's links may route through the deleted node, so most systems only mark it as a tombstone and skip it at query time, leaving the graph progressively riddled with holes. The clean fix is to periodically rebuild the index from scratch, which restores quality but is expensive in both compute and the memory needed to hold two copies during the swap.

Production systems resolve this with a two-tier design that should feel familiar from log-structured storage. New and changed vectors land first in a small, write-friendly fresh segment, often a brute-force or lightly-indexed buffer that is cheap to mutate and is searched in full on every query. Periodically, in the background, that segment is merged into the large, high-quality base index and a fresh empty buffer takes its place. A query searches both tiers and merges the results, so a newly inserted vector is findable the instant it hits the fresh segment, while the bulk of the collection stays in a well-built index. The cost is a small per-query overhead for the brute-force buffer and a steady background rebuild load; the benefit is that freshness and index quality stop being mutually exclusive. This is the same compaction pattern that distributed storage engines use, met in Chapter 8, here adapted to similarity indexes.

7. Choosing and Operating a Vector Store Intermediate

With the architecture and the landscape in hand, the operating decision becomes a short checklist driven by which properties actually bind. If the collection fits in memory, never changes, and lives inside one application, a library is enough and a database is overhead. If vectors are your workload at scale, mutate continuously, and need filtered, available, durable search, a dedicated vector database earns its operational cost. If you already run a relational or search database and the vector collection is secondary, an extension keeps your system count down. Within whichever family you pick, the knobs that matter most are the ANN parameters that trade recall for latency (deepened in Section 25.4), the shard and replica counts that trade memory and availability for cost (in Section 25.5), the freshness policy from Section 6, and the filter strategy from Section 4.

The recurring discipline from Section 1.1 applies here too: distribute because a ceiling binds, not for elegance. A single-node FAISS index serving a million static vectors with low query volume needs none of the machinery in this section, and adding a coordinator and replicas would only buy latency and operational burden. Reach for the distributed vector database when the collection outgrows one machine's memory, when the write rate demands live CRUD, when filtered multi-tenant search is a correctness requirement, or when availability targets forbid a single point of failure. Those are the ceilings; everything in this section is the remedy. The next section opens the shard's black box and shows how the local index answers a query fast in the first place.