Sharded Vector Search

"I am one slice of an index, and I rank only what I am allowed to see. When the coordinator asks, I answer for my shard and forget the rest exists. Somehow, between the four of us, the agent always gets the right ten."

A Vector Index, Sharded Across Nodes That Vote on Relevance

The previous section gave the agent a memory and a planning loop; this section gives it eyes into the enterprise's documents. When the agent decides it needs a fact it does not carry in context, it calls a retrieval tool, and that tool must return the most relevant passages the current user is permitted to read, in tens of milliseconds, from a corpus too large to live on one machine. The retrieval primitives are exactly those Chapter 25 developed for distributed search and Section 36.6 developed for sharded ranking; what changes here is the operating context. A web-scale RAG pipeline retrieves over a public corpus where every document is fair game. An enterprise agent retrieves over a corpus where the same query from two different users must return different results, because they have different permissions, and a single leaked passage is a security incident, not a relevance miss. We build the sharded index first, then layer the permission filter, the hybrid score, the managed-platform option, and the freshness contract on top of it.

1. The Sharded ANN Index Behind the Tool Intermediate

An enterprise corpus of a few hundred million chunks, embedded at 768 or 1024 dimensions in single precision, occupies hundreds of gigabytes before any index structure is added, and the navigable-small-world graphs or inverted lists that make search fast inflate that further. No single node holds it, so the index is partitioned across $S$ shards exactly as Chapter 25 describes in its treatment of index sharding and replication, and as Section 36.6 applies to sharded retrieval. Each shard owns a disjoint slice of the corpus and builds a self-contained approximate-nearest-neighbor structure over its own vectors: a Hierarchical Navigable Small World graph (HNSW) when memory is plentiful and recall must be high, or an inverted file with product quantization (IVF-PQ) when the footprint must be compressed. The two coexist in one cluster, because shards can be tuned independently; Figure 40.4.1 shows both in the same fleet.

A query for the global top-$k$ becomes a scatter-gather, the pattern Section 36.6 builds in detail. The coordinator sends the query vector to every shard, each shard returns its own local top-$k$ candidates with their similarity scores, and the coordinator merges the $S$ lists and keeps the best $k$ overall. The correctness of the merge rests on a simple fact: a globally top-ranked item must be top-ranked within whatever shard happens to hold it. Formally, if shard $s$ returns its top-$k$ set $T_s$ and we keep

then $T$ equals the exact global top-$k$ whenever every globally-best item survived its shard's local cutoff. With exact per-shard ranking that condition always holds for $k$ items, because a shard contributing $m \le k$ of the global winners ranks all of them inside its own top-$k$. The approximation enters only through the ANN search inside each shard, not through the merge, which is why we can reason about recall shard-by-shard. The latency of the whole operation is governed by the slowest shard, not the sum,

so a single straggler shard, the failure mode named in Section 2.7, sets the user-visible response time. Replicating hot shards and hedging requests against the slowest replica is the standard remedy, carried over from Chapter 25's discussion of replication.

2. Permission Filtering at Scale Advanced

Here the enterprise agent departs from the public-corpus RAG of Chapter 36. The agent acts for a user, and that user may read only the documents their access-control list (ACL) permits: their team's wiki, the projects they are staffed on, the customer accounts they own. The retrieval tool must therefore return the top-$k$ relevant items among those the user may see, which is a filtered nearest-neighbor query. Getting this wrong is not a quality regression; it is a data-exfiltration path, and a capable agent will happily summarize whatever you hand it, permissions or not. The filter is a security boundary, and it belongs inside the retrieval cluster, never in a post-hoc check the agent could be prompted around.

The subtlety is where the filter runs. Filtering after the ANN search (post-filtering) is simple but can starve the result: if the user may see one percent of the corpus and the ANN search returns the global top 100, perhaps only one survives, and the user-visible recall collapses. Filtering before the search (pre-filtering) restricts the candidate set to permitted items but can defeat the very graph or inverted-list structure that makes ANN fast, because the reachable neighborhood is now sparse. Production systems take a middle path: they fold the ACL predicate into the ANN traversal itself, so the search walks the graph but only accepts permitted nodes as candidates, and they over-fetch (raise the per-shard $k$) to keep enough permitted items in flight. The recall the user actually experiences is a filtered quantity,

where $G_k(q,u)$ is the exact top-$k$ over the user-visible subset and $R_k(q,u)$ is what the sharded system returned. Two retrievals for the same query against the same index can have different $G_k$, because $\text{visible}(x,u)$ depends on the user. The demonstration in Code 40.4.1 builds a sharded index with per-item ACL tags, runs the filtered scatter-gather, and checks that the result equals $G_k$ exactly and never includes a forbidden item.

The code below constructs $S$ shards over a synthetic corpus, tags each item with the groups allowed to read it, and answers a query for a user in two groups. Each shard applies the ACL predicate to its own candidates before ranking, then returns its local top-$k$; the coordinator merges. We verify the merged result against a brute-force exact filtered top-$k$ and confirm that no item the user may not see ever appears.

import numpy as np

# A tiny sharded vector index with per-item ACL tags. Each item belongs to one
# shard and carries a set of group ids the viewer must intersect to be allowed.
rng = np.random.default_rng(7)
N, d, S = 6000, 32, 4               # items, embedding dim, shards
groups = [f"g{i}" for i in range(6)]

vecs = rng.standard_normal((N, d)).astype(np.float64)
vecs /= np.linalg.norm(vecs, axis=1, keepdims=True)        # cosine == dot product
# Each item is readable by a random subset of 1-2 groups.
acl = [set(rng.choice(groups, size=rng.integers(1, 3), replace=False)) for _ in range(N)]
shard_of = rng.integers(0, S, size=N)                      # static placement

# The querying agent acts on behalf of a user in these groups.
user_groups = {"g1", "g4"}
def visible(i):                                            # ACL predicate
    return len(acl[i] & user_groups) > 0

q = rng.standard_normal(d); q /= np.linalg.norm(q)
k = 10

# ---- Exact filtered top-k (ground truth): score every VISIBLE item globally.
allow = np.array([visible(i) for i in range(N)])
scores = vecs @ q
masked = np.where(allow, scores, -np.inf)
exact = np.argsort(-masked)[:k]
exact_set = set(int(i) for i in exact)

# ---- Sharded scatter-gather: each shard filters by ACL FIRST, then ranks its
# own visible items, returns its local top-k; a coordinator merges the S lists.
def shard_topk(sid):
    idx = np.where(shard_of == sid)[0]
    idx = idx[[visible(int(i)) for i in idx]]              # ACL filter on the shard
    if idx.size == 0:
        return []
    s = vecs[idx] @ q
    order = np.argsort(-s)[:k]
    return [(int(idx[j]), float(s[j])) for j in order]

partials = [shard_topk(sid) for sid in range(S)]           # S parallel workers
merged = sorted(sum(partials, []), key=lambda t: -t[1])[:k]
merged_set = set(i for i, _ in merged)

print("items / shards / dim     :", N, S, d)
print("viewer groups            :", sorted(user_groups))
print("visible items (of N)     :", int(allow.sum()))
print("exact filtered top-k     :", sorted(exact_set))
print("sharded gather top-k     :", sorted(merged_set))
print("recall@k vs exact        :", f"{len(exact_set & merged_set)/k:.2f}")
print("any forbidden item leaked:", any(not visible(i) for i in merged_set))

items / shards / dim     : 6000 4 32
viewer groups            : ['g1', 'g4']
visible items (of N)     : 2861
exact filtered top-k     : [238, 305, 2260, 2608, 2874, 3376, 3499, 4069, 4522, 5706]
sharded gather top-k     : [238, 305, 2260, 2608, 2874, 3376, 3499, 4069, 4522, 5706]
recall@k vs exact        : 1.00
any forbidden item leaked: False

3. Hybrid Lexical and Dense Retrieval Intermediate

Dense embeddings capture meaning but miss exact tokens: an account number, an error code, a product SKU, a person's surname. An enterprise agent asked to "find the ticket where customer ACME-4471 reported error E2207" needs those literals matched, not paraphrased. Hybrid retrieval, developed for the distributed case in Chapter 25's section on distributed hybrid search, runs a lexical index (BM25 over an inverted index) alongside the dense ANN index and fuses their rankings. The fusion is typically reciprocal rank fusion or a weighted score combination,

where the two component scores are normalized to a common range and $\alpha$ trades semantic recall against literal precision. In a sharded deployment both indexes are partitioned the same way, so each shard contributes a fused local top-$k$ and the scatter-gather of Section 1 carries over unchanged; the only addition is that each shard now runs two retrievers and fuses before it returns. The permission filter of Section 2 applies identically to both, because the ACL is a property of the item, not of how it was scored. For the agent, hybrid retrieval is what makes the tool dependable across both "explain our refund policy" (dense wins) and "pull invoice INV-99812" (lexical wins) without the user choosing a mode.

4. Build Versus Buy: Managed Vector Databases Beginner

Everything in Sections 1 through 3, sharded ANN, scatter-gather, ACL filtering, hybrid fusion, replication, and freshness, is offered as a managed service by vector databases and the managed platforms of Section 33.9. Pinecone, Weaviate, Qdrant, Milvus, and the vector extensions of the cloud data platforms expose a single query endpoint that hides the shards, runs the filtered ANN, and handles the index updates. The build-versus-buy decision is the enterprise version of the right-tool principle that runs through this book. Building your own gives control over the ANN parameters, the filter semantics, and the placement, and avoids per-query egress pricing at very high volume; buying gives you an SLA, automatic replication and rebalancing, and metadata-filter support you do not have to make correct yourself, which for a security-critical filter is a meaningful reduction in risk. For most agentic applications the managed path is the right first move, because the differentiating work is the agent, not the reimplementation of a sharded index that a vendor has already hardened. The library shortcut below shows how thin the application-facing query becomes when the platform owns the distribution.

# Option A: FAISS, you own the shard, the library owns the ANN structure.
import faiss, numpy as np
index = faiss.IndexHNSWFlat(d, 32)          # HNSW graph, M=32 neighbors
index.add(vecs.astype('float32'))           # build this shard's index
scores, ids = index.search(q[None].astype('float32'), k)   # local top-k

# Option B: a managed vector DB owns the shards, the ANN, AND the filter.
# The metadata filter is the security boundary; the service applies it
# inside the sharded search, not after it.
result = index_client.query(
    vector=q.tolist(),
    top_k=k,
    filter={"acl_groups": {"$in": ["g1", "g4"]}},   # only items the user may read
)

5. Freshness and Consistency as the Index Updates Advanced

An enterprise corpus is not static. Documents are created, edited, and deleted; permissions change when someone joins a team or a project is reclassified. The index must reflect these changes, and the agent expects a degree of freshness: a document uploaded a minute ago should be retrievable, and a document the user just lost access to must vanish from their results immediately, because a stale ACL is a leak. These are the consistency questions of Section 2.5 applied to the retrieval layer. Most vector systems are eventually consistent for content: a write enters a fast mutable segment, becomes searchable within seconds, and is later merged into the compacted immutable index, so newly added vectors are visible before the background reindex completes. For permission revocations the looser guarantee is unacceptable, so production systems evaluate the ACL against a strongly-consistent authorization store at query time rather than trusting the possibly-stale group tags baked into the index, accepting a small extra lookup to close the revocation gap. The split is deliberate: eventual consistency for which documents exist, strong consistency for who may read them.

The tool is now complete: a sharded, hybrid, permission-filtered, fresh vector index that returns the right passages for the right user in a single call. The next section connects it to generation. The retrieved passages become the context a distributed LLM serving stack conditions on, turning retrieval into retrieval-augmented generation for the enterprise agent; that consumer is built in Section 40.5, and Section 40.6 shows the agent calling this index as one tool among several in its reasoning loop.