In my previous post on the economics of late interaction retrieval, I made the case that PLAID and MUVERA had fundamentally changed the cost calculus for multi-vector search, taking late interaction from a “hard no” to a “viable tradeoff” at production scale. I stand by that analysis—for the specific problem they were designed to solve.

But there’s a subtlety worth unpacking. When we talk about “multi-vector retrieval,” we’re actually talking about a much larger family of problems than the ColBERT-MaxSim scenario that dominates the conversation. And when you zoom out from that specific case, some uncomfortable truths emerge:

1. The most hyped algorithms are geometry-dependent—they assume a very specific structure in how embeddings are distributed, and that structure is not universal
2. The dominant similarity function (MaxSim) is one of many—and the others don’t always cooperate with PLAID or MUVERA’s machinery
3. Some “hard” multi-vector problems have embarrassingly simple solutions that don’t require any specialized infrastructure at all

The argument runs in three acts. First, we’ll establish the two axes that determine which infrastructure choice is right for any given problem: the similarity function and the dataset geometry. Then we’ll apply both axes to stress-test PLAID and MUVERA—not to dismiss them, but to be precise about where their assumptions hold and where they don’t. Finally, we’ll show that two “hard” multi-vector problems have simple closed-form solutions hiding inside those same assumptions, with immediate practical implications for how you architect retrieval systems today. If you want to skip ahead, the decision framework table at the end of the post is the practical output.

Let’s dig in.

The General Case of Set-to-Set Similarity

In the standard neural retrieval setting, you’re doing point-to-point Maximum Inner Product Search (MIPS): a single query vector against a collection of single document vectors. The similarity is simply:

This maps directly to an ANN index problem, and the entire ecosystem—HNSW, IVF, DiskANN, Product Quantization—is built to solve it efficiently.

Multi-vector retrieval generalizes this to set-to-set similarity: each entity is a set of embedding vectors $Q = {q_1, \ldots, q_m}$ and $P = {p_1, \ldots, p_n}$, and no single dot product tells you how similar they are. What you need is an aggregation function over all pairwise interactions.

Here’s where things get interesting—because there are many such aggregation functions, and they’re used in real applications:

MaxSim (Chamfer Similarity): The ColBERT standard. For each query token, find the best-matching document token; sum these maximal matches.

This is the workhorse of late interaction retrieval. It allows fine-grained token alignment—each query token finds its best match independently—and it’s asymmetric (the sum is over query tokens, not document tokens).

SumSim (All-Pairs): Sum all pairwise similarities.

This treats every query-document token pair as equally relevant. It appears in cross-modal matching and group recommendation settings where you want aggregate affinity rather than fine-grained alignment.

Top-K Sum: For each query token, sum the top-$k$ (rather than just the maximum) document similarities.

This is a softer variant of MaxSim that rewards documents where multiple tokens strongly match each query token, not just one. Useful when entity sets have structured redundancy.

Symmetric Chamfer: Average of MaxSim in both directions—from query to document and from document to query.

Used in some image-text matching settings where neither modality “anchors” the comparison.

Just this brief survey should give you pause. The literature discussing PLAID and MUVERA as solutions to “multi-vector retrieval” is almost entirely talking about MaxSim, and treating MaxSim as if it were synonymous with the problem itself. That’s a significant conflation—and as we’ll see, the algorithms break down in revealing ways when you change the scoring function or the data distribution.

Dataset Geometry: What Are We Actually Indexing?

The second axis of analysis—equally important and equally underappreciated—is dataset geometry: the statistical structure of how document token embeddings are distributed in embedding space.

Three Geometric Regimes

There are three primary distributional patterns to understand:

Gaussian Isotropic: Embeddings are drawn from a single distribution with uniform variance in all directions. This is the world that most ANN theory assumes—a “well-behaved” embedding space where random partitioning methods (like SimHash) work with provable guarantees.

Here $\mathbf{v} \in \mathbb{R}^d$ is a single token embedding vector, $\boldsymbol{\mu} \in \mathbb{R}^d$ is the distribution mean (the centroid of the cloud), $\sigma^2$ is a single scalar variance applying equally in every direction, and $\mathbf{I}$ is the $d \times d$ identity matrix—meaning no direction in embedding space is preferred over any other.

Gaussian Anisotropic: Embeddings are still unimodal but have unequal variance across dimensions—stretched or compressed in specific directions. BERT’s raw sentence embeddings, for instance, are famously anisotropic, with a few dominant principal components capturing most of the variance. This breaks the isotropy assumptions of many hashing-based methods.

The key change is that the scalar $\sigma^2$ is replaced by a full covariance matrix $\boldsymbol{\Sigma} \in \mathbb{R}^{d \times d}$. When $\boldsymbol{\Sigma}$ has unequal eigenvalues, the distribution is stretched along some dimensions and compressed along others—producing an ellipsoidal cloud rather than a spherical one. The constraint $\boldsymbol{\Sigma} \neq \sigma^2 \mathbf{I}$ simply means the variances are not identical in all directions.

Multi-Kernel Anisotropic: Embeddings are drawn from a mixture of anisotropic Gaussians, where each mixture component (“kernel”) corresponds to a semantic cluster, and each kernel can have its own covariance structure.

This extends the anisotropic case to $K$ distinct clusters. Each cluster $k$ has its own centroid $\boldsymbol{\mu}_k$ and its own covariance matrix $\boldsymbol{\Sigma}_k$ (so different clusters can have different shapes and orientations). The mixing weights $\pi_k \geq 0$ give the probability that a randomly drawn token embedding comes from cluster $k$; the constraint $\sum_k \pi_k = 1$ ensures these probabilities are well-formed. In practice, $\pi_k$ reflects how often a semantic category (financial terms, function words, visual elements) appears in the corpus.

This third regime is the realistic one for trained multi-vector embeddings like ColBERTv2. The token embeddings from a trained language model don’t form a single Gaussian blob—they cluster by semantic meaning. All token embeddings for terms related to “financial instruments” cluster in one region of space; “biological processes” in another; function words (“the”, “and”, “is”) in yet another. Each cluster has its own orientation (some are compact and round, others are elongated along specific axes corresponding to the semantic variation within that topic).

Three distributional regimes: (left) Gaussian Isotropic—uniform variance in all directions; (center) Gaussian Anisotropic—stretched in specific directions; (right) Multi-Kernel Anisotropic—the realistic regime for trained ColBERT embeddings, with distinct semantic clusters each having their own shape and orientation. PLAID exploits this structure via its centroid codebook; MUVERA’s SimHash assumes the first regime and struggles with the third.

Why Multi-Kernel Anisotropic? The Contrastive Training Connection

This isn’t coincidence. The multi-kernel anisotropic structure is a direct consequence of how ColBERT and its family are trained using contrastive objectives.

In contrastive training, the model learns to pull positive query-document pairs together in embedding space while pushing negatives apart. The loss function that formalizes this is called InfoNCE—a name that carries meaning worth unpacking.

“NCE” stands for Noise Contrastive Estimation, a technique originally designed to train language models without computing an expensive softmax over a vocabulary of millions of words. The insight is to reframe the problem as binary classification: teach the model to distinguish real (signal) data from noise (randomly sampled negative) data, rather than predicting the exact right answer from all possible answers at once. This sidesteps the normalization constant that makes full softmax expensive. “Info” indicates that this particular loss is motivated as a lower bound on mutual information between representations—maximizing InfoNCE is equivalent to maximizing a variational lower bound on $I(\text{query}; \text{positive document})$. This information-theoretic grounding, introduced by van den Oord et al. (2018) in the context of sequential prediction, is what distinguishes InfoNCE from a plain cross-entropy or triplet loss.

The practical upshot: unlike standard cross-entropy (which needs a fixed, known label space) or MSE (which needs continuous target values), InfoNCE works wherever you can define positive/negative pairs—which is exactly the supervised retrieval setting. The loss takes the form:

where $q$ is a query embedding, $p^+$ is its positive (relevant) document embedding, $\mathcal{N}$ is a set of negative (non-relevant) documents, and $\tau$ is a temperature parameter. The numerator rewards similarity between $q$ and its true match; the denominator forces the model to simultaneously push down similarity to all negatives. Minimizing this loss is equivalent to making the positive pair score stand out from the crowd of negatives—directly shaping the relative geometry of the embedding space.

Under this objective, the geometry that emerges is predictable. The temperature $\tau$ acts as a “cluster tightness” parameter: lower temperature concentrates embeddings into tighter clusters; higher temperature allows more spread. The training process creates equi-angular clusters—distinct groups separated radially on the embedding hypersphere, with positive pairs clustered tightly within each group.

But the key insight for multi-vector retrieval is what happens within a document. A document about financial derivatives contains tokens spanning multiple semantic roles: financial terminology (“derivative,” “swap,” “yield”), syntactic function words (“the,” “of,” “which”), and contextual modifiers (“complex,” “structured,” “underlying”). Each of these semantic roles corresponds to a different region of the embedding space—a different kernel—and the tokens from a single document are distributed across these kernels.

This is the “multi-kernel” structure. And the “anisotropy” comes from the fact that within each kernel, the variance is not spherical. Hard negative mining—a standard component of ColBERT-style training—specifically shapes the within-kernel geometry in a way that needs careful explanation.

A hard negative is a document that the model currently finds hard to reject: it is non-relevant (a true negative by ground-truth label) but has an embedding that is close to the query in the current embedding space—similar wording, overlapping topic, or superficially matching structure. The “hard” refers to the model’s difficulty of discrimination at training time, not to the documents being inherently similar in meaning. This contrasts with easy negatives—randomly sampled documents that are obviously unrelated and produce near-zero gradient, providing no useful learning signal.

Hard negative mining surfaces these ambiguous near-misses and forces the model to push them apart from positives. Because the negatives are semantically proximate to the query (same broad topic, similar surface form), this pressure acts within a semantic neighborhood rather than across the full embedding space. The result is directional: the model learns to stretch the representation along axes that discriminate fine-grained relevance within a topic cluster, while keeping inter-cluster distances large. This directional stretching within semantic clusters is precisely what produces elongated, non-spherical within-kernel covariance.

The hard-negative-augmented loss is:

where $\mathcal{H}$ are hard negative passages—non-relevant documents that are superficially close to the query (retrieved by BM25 or an earlier model version, but labelled non-relevant). Because these hard negatives are drawn from the same semantic neighborhood as the query, the gradient of this loss explicitly pushes the model to create directional distinctions within semantic neighborhoods rather than between them—stretching the within-kernel distribution along the discrimination axis rather than compressing it uniformly.

Quantifying the Geometry: Intra-Document Variance and Residuals

Before examining where PLAID and MUVERA break down—and before the two simplifications that are the practical payoff of this analysis—we need two concrete diagnostic quantities that connect geometry to algorithmic behavior.

Intra-document variance ($\sigma^2_{\text{intra}}$) measures how spread out a document’s token embeddings are around their centroid:

High \(\sigma^2_{\text{intra}}\) means a document’s tokens are spread across many semantic regions—many distinct kernels are activated per document. Low $\sigma^2_{\text{intra}}$ means tokens cluster tightly around a single centroid.

Residual magnitude ($|r|$) measures the reconstruction error after assigning each token to its nearest codebook centroid $c^*$:

A small $|r|$ means the centroid codebook provides a good approximation; a large $|r|$ means the codebook is missing significant structure in the data.

These two quantities—$\sigma^2_{\text{intra}}$ and $|r|$—are the key indicators for predicting how well PLAID and MUVERA will perform on a given dataset. Let’s see why.

A document with low intra-document variance (left) has token embeddings tightly clustered around a single centroid—typical of a homogeneous short text passage. A document with high intra-document variance (right) has tokens spread across multiple distinct semantic kernels—typical of a complex multi-topic page like a ColPali image with mixed text, charts, and diagrams. PLAID’s centroid pruning is highly selective in the left case; nearly ineffective in the right.

PLAID Under a Geometric Microscope

As I covered in detail in the previous post, PLAID’s insight is to use a centroid codebook to prune the candidate set early. The centroid inverted list tells you which documents have tokens assigned to which centroids, and you can eliminate documents whose centroids don’t match query token embeddings—without ever reading the expensive full-residual representations from disk.

This is elegant, and it works beautifully for the ColBERTv2 MS MARCO case. But the efficiency of this pruning depends critically on the geometry.

How Intra-Document Variance Determines Selectivity

PLAID’s document selectivity—the fraction of documents eliminated by centroid pruning—depends on how many distinct centroids each document activates. Let $n_d$ be the number of distinct centroids across all tokens in document $d$. The probability that document $d$ survives an initial candidate generation sweep (i.e., has at least one token centroid matching a query centroid) is approximately:

where $C$ is the total codebook size (e.g., $C = 2^{16} = 65{,}536$ for ColBERTv2) and $n_{\text{query}}$ is the number of distinct centroids activated by the query (roughly equal to the number of query tokens, $|Q|$).

For the ColBERTv2 MS MARCO case, the centroid math works strongly in PLAID’s favor. MS MARCO passages average roughly 55–80 tokens (with doc_maxlen=180 as a ceiling); the default codebook size is $C = 2^{16} = 65{,}536$. Semantically similar tokens are assigned to the same centroid by design, so the number of distinct centroids activated per passage $n_d$ is substantially smaller than the token count—many function words, common phrases, and topically related terms cluster together. The PLAID paper (Santhanam et al., 2022) empirically validates this indirectly: centroid-only retrieval achieves >99% recall for top-$k$ passages within just $10k$ candidates across 8.8M passages on MS MARCO, which is only possible because the average passage activates a very small fraction of the 65,536-centroid codebook. If passages activated many hundreds of distinct centroids, the centroid-only stage would produce far more candidates to reach the same recall threshold.

Now consider what happens when $\sigma^2_{\text{intra}}$ is high—documents whose tokens span many different semantic regions. If $n_d$ grows substantially (approaching or exceeding several hundred distinct centroids per document), the survival probability above approaches 1 for nearly every document in the corpus. PLAID’s centroid pruning stage eliminates almost nothing. You’re left paying for Stage 4 exact scoring on the full candidate set, which collapses to the naive baseline.

This is not a hypothetical—it’s a real concern for ColPali embeddings of complex document pages, where a single page may contain text in multiple languages, diagrams, charts, captions, and decorative elements, each mapping to different semantic kernels. The intra-document variance for a complex financial PDF page is expected to be substantially higher than for a homogeneous text passage.

How the Residual Magnitude Determines Quantization Quality

The second failure mode for PLAID is more subtle but equally important: the quality of the 2-bit residual compression degrades as the residual magnitude grows.

Recall that ColBERTv2 compresses each token by storing the centroid ID plus a quantized residual $r_q$. The reconstruction error for a token $p$ assigned to centroid $c^*$ is:

This error grows with $|r| = |p - c^*|$. When the residual is large—meaning the codebook centroids are a poor fit for the actual token distribution—aggressive 2-bit quantization introduces substantial reconstruction noise. The approximate MaxSim scores computed during PLAID’s Stage 3 become less reliable, which in turn means more false positives survive to Stage 4, increasing the expensive exact-scoring burden.

The codebook (left) divides embedding space into semantic regions, each with a centroid c. When a token p falls near its assigned centroid (center panel), the residual r = p − c* is small and 2-bit quantization introduces minimal reconstruction error—p̂ ≈ p. When p is far from any centroid (right panel), the residual is large and deQuantization produces an approximation p̂ with significant drift—degrading the Stage 3 scoring fidelity in PLAID.*

Formally, the MaxSim approximation error can be bounded as:

where $\hat{q}_i$ is the reconstructed approximation of query token $q_i$. When residuals are large (poor codebook fit), these reconstruction errors compound, and the bound loosens—meaning the centroid interaction scores in Stage 3 are less trustworthy as proxies for exact MaxSim.

How SumSim and Top-K Sum Break PLAID’s Selectivity

This is where the set-to-set similarity function becomes a first-class concern.

PLAID’s Stage 2 (centroid pruning) works by reasoning about MaxSim: if a centroid $c_k$ has low similarity to all query tokens, it cannot contribute to the MaxSim score, so it can be safely pruned. The pruning logic is:

This is sound for MaxSim because the max operation means that a centroid that doesn’t strongly match any query token can never be the “winning match” for any query token. The contribution of centroid $c_k$ to the final score is bounded by $\max_i \langle q_i, c_k \rangle$.

Now replace MaxSim with SumSim:

Under SumSim, every document token contributes to every query token’s score—there’s no “winner takes all.” A centroid with low similarity to all query tokens individually may still be part of a document that has high total aggregate similarity. The safe pruning condition for MaxSim no longer holds. In fact, any centroid with a non-zero dot product with any query token must be considered, which in practice means centroid pruning eliminates essentially nothing.

The same argument applies to Top-K Sum, where the contribution of a document token is capped at being in the top-$k$ matches per query token rather than strictly the maximum. The mathematical boundary on the residual’s contribution to the aggregate score is looser:

This larger bound means PLAID’s pruning threshold must be set proportionally lower to remain safe, which in turn means fewer candidates get pruned. For large $k$, the advantage essentially vanishes.

The Bottom Line for PLAID: PLAID is purpose-built for MaxSim scoring on multi-kernel anisotropic data where intra-document variance is moderate and residuals are small. When either condition fails—high variance, large residuals, or a non-MaxSim similarity function—the algorithm’s guarantees weaken and its practical efficiency degrades.

MUVERA Under a Geometric Microscope

MUVERA’s approach is mathematically different from PLAID’s: rather than building a custom inverted index for multi-vector data, it transforms multi-vector sets into fixed-dimensional vectors (FDEs) via SimHash partitioning, enabling standard single-vector MIPS. The theoretical promise is an $\epsilon$-approximation guarantee for the Chamfer similarity (MaxSim).

But the theory comes with an assumption baked in: that the SimHash partitioning correctly places nearby vectors into the same bucket. Let’s examine what happens when this assumption breaks down.

SimHash and the Anisotropy Problem

SimHash partitions $\mathbb{R}^d$ using $b$ random hyperplanes ${h_1, \ldots, h_b}$, each drawn independently and uniformly from the sphere $\mathbb{S}^{d-1}$. Two vectors $u, v$ are assigned to the same partition if $\text{sign}(\langle h_l, u \rangle) = \text{sign}(\langle h_l, v \rangle)$ for all $l = 1, \ldots, b$.

The probability that $u$ and $v$ share a partition is exactly:

where $\theta_{uv} = \arccos\left(\frac{\langle u, v \rangle}{|u||v|}\right)$ is the angle between them. This monotonically decreases in $\theta_{uv}$ and in $b$—as vectors become more similar (smaller angle) and as you use fewer hyperplanes, collision probability increases.

For isotropic data, random hyperplanes are an excellent partitioning strategy because no direction is privileged; the hyperplanes respect the underlying geometry.

For anisotropic data, the situation is different. Consider a kernel (semantic cluster) with a covariance matrix $\boldsymbol{\Sigma}_k$ that is elongated—much larger variance along a specific direction $\mathbf{u}$ than orthogonal directions:

Within this kernel, two vectors $p_1, p_2$ that are nearby in $\ell_2$ distance may have a relatively large angular separation $\theta_{p_1 p_2}$, because the elongation along $\mathbf{u}$ produces many pairs with large directional displacement but small absolute distance.

Now a random hyperplane $h$ that is perpendicular to $\mathbf{u}$—which is not unlikely, given random sampling—will split the cluster along its major axis, separating $p_1$ and $p_2$ into different buckets despite their proximity. This directly reduces the probability that a query token and its best-matching document token share a SimHash partition.

The practical consequence: in a multi-kernel anisotropic space, the effective $P(\text{same partition})$ for true nearest-neighbor pairs is substantially lower than the theoretical prediction from cosine similarity alone. This means MUVERA’s FDE dot product systematically underestimates Chamfer similarity, increasing recall error. (To my knowledge, this degradation has not been directly quantified on ColBERT-family embeddings as a function of within-kernel anisotropy—it would be a clean empirical contribution.)

SimHash failure with anisotropic data. Left: in an isotropic (spherical) cluster, a random hyperplane is unlikely to separate nearby vectors q and p—they land in the same bucket and the FDE captures the correct match. Right: in an elongated anisotropic cluster trained with hard negatives, a hyperplane perpendicular to the major axis splits q and p* into different buckets despite their small ℓ₂ distance. The effective angular separation θ_eff inflated by anisotropy is what drives the collision probability down—this is the root cause of MUVERA’s recall degradation on ColBERT-family embeddings.*

The Over-Partitioning Problem

There’s a second failure mode for MUVERA that operates in the opposite direction: using too many hash bits $b$ (too many partitions) in an anisotropic setting.

In MUVERA’s FDE construction, the total number of partitions is $K = 2^b$. The FDE dimension grows as $r \times K \times d_{\text{proj}}$, so practitioners balance partitioning fineness against FDE size. But increasing $b$ has a subtle secondary effect: with finer partitioning on anisotropic data, the probability that the same pair of similar vectors land in the same bucket decreases faster than on isotropic data.

Formally, for an anisotropic cluster with effective angle $\theta_{\text{eff}} > \theta_{uv}$ (the “apparent” angular separation inflated by anisotropy), the collision probability becomes:

which decays exponentially in $b$. As $b$ grows, the effective recall of MUVERA’s candidate generation phase degrades—not from the FDE being uninformative, but from the partitioning not placing nearest neighbors into the same bucket reliably.

There is therefore a trade-off: too few partitions ($b$ small) gives coarse FDEs with poor discrimination between non-relevant documents; too many partitions ($b$ large) causes correct nearest-neighbor pairs to be separated, reducing recall. For isotropic data, this trade-off has a natural sweet spot that’s relatively easy to tune. For anisotropic data—and especially multi-kernel anisotropic data—the window is narrower and the optimum shifts in ways that depend on the specific covariance structure of each kernel.

The Bottom Line for MUVERA: MUVERA’s theoretical guarantees assume that SimHash correctly partitions the embedding space such that nearest neighbors share buckets with high probability. In multi-kernel anisotropic spaces—which is what trained ColBERT-family models produce—this assumption is violated for elongated within-kernel distributions. The failure is exacerbated by increasing the number of partitions.

Hacks: Where Elaborate Infrastructure Isn’t Needed

Now for the part I find genuinely exciting—two cases where the “hard” multi-vector problem turns out to be trivially reducible to a problem we already know how to solve.

Hack #1: SumSim Is Just Mean-Pooled MIPS

At first glance, SumSim seems to require multi-vector infrastructure: you have sets $Q$ and $P$, and you need to sum over all $|Q| \times |P|$ pairwise dot products. Surely this requires something like PLAID or MUVERA?

Let’s do the algebra:

By the bilinearity of the inner product:

where $\bar{q}$ and $\bar{p}$ are the mean-pooled embeddings of the query and document sets, respectively.

When comparing documents for a fixed query, $|Q|$ is constant and can be dropped. If document sets also have fixed size $|P|$ (as with ColPali’s 1030 patches), the prefactor is a constant. We are left with:

This is a standard MIPS problem. You can compute mean-pooled representations offline for every document, index them in any standard ANN system (HNSW, DiskANN, ScaNN), and retrieve them with a single mean-pooled query vector. No PLAID. No MUVERA. No custom infrastructure whatsoever.

This has immediate practical implications. Any retrieval system where the scoring function is SumSim—aggregate affinity, group recommendation, cross-modal matching with all-pair scoring—does not need multi-vector retrieval infrastructure. Mean pooling and a standard vector database is the correct solution, and it’s orders of magnitude cheaper.

The mathematical equivalence also provides a clear intuition for when SumSim is the right choice: when every token-to-token interaction is genuinely informative, with no winner-takes-all dynamics. For dense, topic-homogeneous queries and documents, this can be reasonable. For queries where only a few tokens carry the semantic signal—typical in information retrieval—MaxSim’s “find the best match” semantics are more appropriate.

The four-step reduction: (1) SumSim requires |Q|×|P| pairwise dot products—superficially a multi-vector problem; (2) bilinearity of the inner product collapses these into a single dot product of summed vectors; (3) factoring out the set sizes gives the mean-pooled dot product ⟨q̄, p̄⟩ up to a constant; (4) this is a standard MIPS problem—index p̄ per document in any ANN system and query with q̄. No PLAID. No MUVERA. O(log N) lookup.

Practical implication: If you’re using SumSim and running PLAID or MUVERA, you’re paying for infrastructure you don’t need. Mean-pool your embeddings, use a standard HNSW index, and call it done.

Hack #2: ColPali Doesn’t Need What ColBERT Needs

ColPali generates exactly 1030 patch embeddings per page (a fixed 32×32 grid of 1024 image patches plus 6 instruction text tokens), each projected into a 128-dimensional space. This fixed structure is fundamentally different from ColBERTv2 text embeddings, where document length varies—a 20-token passage might have 20 embeddings; a 300-token passage has 300.

This variability in ColBERT text is precisely what makes indexing challenging. You can’t pre-aggregate to a fixed representation because you don’t know how many tokens each document will have, and the token count itself is semantically meaningful (document length affects the MaxSim score normalization).

With ColPali, this problem evaporates. Every page has exactly 1030 patches. This opens up indexing strategies that simply aren’t available for variable-length ColBERT:

The Vespa Approach: Per-Token Nearest Neighbor Search. Vespa’s production ColPali implementation indexes each patch embedding as an individual vector in an HNSW index, then at query time, issues one nearestNeighbor query per query token—retrieving the $k$ closest pages per query token using binary (Hamming) embeddings as a first pass. This multi-phase pipeline looks like:

1. Phase 0 (pre-filtering): For each of $|Q|$ query tokens, retrieve $k_0$ nearest page-patches using Hamming distance on binarized embeddings. Union the corresponding pages.
2. Phase 1 (approximate MaxSim): Score surviving pages with approximate MaxSim using inverted Hamming distance. Keep top $k_1$ pages.
3. Phase 2 (exact MaxSim): Rerank the top $k_1$ pages using full-precision float MaxSim.

With binarized patch embeddings (128 bits = 16 bytes per patch, a 32× compression from float32), the per-query cost scales as $|Q| \times k_0 \times O(\log N)$ for the HNSW lookups—very manageable.

The complete ColPali retrieval pipeline. Offline: pages are encoded into 1030 patch embeddings each, binarized to 16 bytes/patch (32× compression), and all patches from all pages are stored in a single shared HNSW index keyed by Hamming distance. Online: each query token independently issues a nearest-neighbor lookup against the shared index (Phase 0), the resulting page sets are unioned, scored with approximate Hamming-based MaxSim (Phase 1), and the top candidates are re-ranked with exact float MaxSim loaded from disk (Phase 2). The fixed patch count of 1030 per page is what makes the per-token HNSW strategy tractable—variable-length ColBERT text cannot exploit this structure.

The Fixed-Centroid Alternative. Because the number of patches per document is fixed at 1030, you can pre-cluster each document’s patches into a fixed number of centroids ($C_d$, say 10-30) using k-means, and index only those centroids. This creates a compact two-level structure:

• Level 1: $C_d$ centroids per document in a standard HNSW index (fast candidate generation)
• Level 2: Full 1030 patch embeddings per document on disk (for exact MaxSim scoring of top candidates)

Candidate generation now requires $O(|Q| \times \log(N \times C_d))$ operations—proportional to the number of query tokens and the log-size of the centroid index. Exact MaxSim is only computed for the top-$k$ candidates, involving $|Q| \times 1030$ dot products per candidate.

This is essentially the PLAID architecture—but with a key difference. Because every document has exactly 1030 patches (not a variable number), the codebook can be constructed per-document rather than globally, and the indexing structure can leverage the fixed cardinality to avoid the centroid-ID allocation overhead that PLAID requires at corpus scale.

The Cost Comparison. For ColPali at 1 billion pages, with binarized patch embeddings:

The binarized approach is the practical winner: 32× compression with minimal recall loss (binarization results in a small drop on DocVQA, recoverable in the float re-ranking phase), and compatibility with Hamming-distance HNSW which is natively supported in systems like Vespa.

Practical implication: ColPali’s fixed patch count of 1030 per page is a gift, not a constraint. It enables indexing strategies based on fixed-cardinality multi-vector representations that don’t exist for variable-length ColBERT text, and makes brute-force MaxSim feasible at smaller scales (tens of millions of pages) without any specialized infrastructure.

Pulling It Together: A Decision Framework

The taxonomy that emerges from this analysis is more nuanced than “use PLAID for late interaction” or “use MUVERA if you want standard infrastructure.” The right architecture depends on both the similarity function and the dataset geometry.

The common thread in the “hacks”: when the structure of your problem is more constrained than the general case PLAID/MUVERA were designed for, simpler solutions often dominate.

Implications for Embedding Model Design

The geometry insights above have a direct design implication that’s often overlooked: you can train for a geometry that’s more indexing-friendly.

In the previous post on domain adaptation, we discussed how domain-specific fine-tuning reshapes the embedding space. The same techniques that tune vocabulary coverage also reshape the geometric structure—and we can be deliberate about it.

For applications where you know the similarity function is MaxSim (the ColBERT case), moderate intra-document variance is desirable: enough spread to achieve good recall in PLAID’s centroid lookup, but compact enough to make Stage 2 pruning effective. You can steer this via the training objective.

One approach is to add an intra-document variance regularizer to the contrastive loss:

where:

By setting $\sigma^2_{\text{target}}$ to a value that empirically yields good PLAID selectivity on your codebook size, you bake indexing efficiency into the embedding model itself.

Similarly, for applications using MUVERA in anisotropic spaces, you might regularize for lower intra-kernel anisotropy—encouraging rounder cluster shapes that play better with random hyperplane partitioning:

This is, admittedly, more involved—it requires estimating per-cluster covariance during training, which is non-trivial. But the point stands: embedding model design and indexing infrastructure aren’t independent choices. The geometry your model learns directly determines how well your index performs, and you have more control over that geometry than is commonly appreciated.

A Note on What This Doesn’t Change

Before closing, it’s worth being precise about what this analysis does and doesn’t refute.

PLAID and MUVERA are genuinely impressive. For the ColBERT-MaxSim-on-MS MARCO problem they were designed for, they represent major advances, and the economic analysis from the previous post holds: late interaction is now viable at scales where it previously wasn’t. Nothing in this post undercuts that.

MaxSim on ColBERT-style text data is the dominant use case, and on that use case, PLAID works well. The failure modes I’ve described are real but they’re edge cases for text retrieval—they become primary concerns for multi-modal retrieval (ColPali), non-standard similarity functions, or domain-specific data with unusual geometric properties.

The “hacks” are not always hacks. SumSim may genuinely be the right scoring function for some applications—my point is only that if it’s your function, you don’t need specialized infrastructure. And ColPali’s fixed patch structure is a deliberate design choice with real advantages; its indexing simplicity is one of those advantages.

What I hope this analysis provides is a more precise vocabulary for thinking about multi-vector retrieval: similarity function and dataset geometry as the two fundamental axes, with infrastructure choices following from them rather than preceding them.

Looking Forward

The questions this analysis raises are, I think, more interesting than the answers it provides:

Can we build geometry-aware hashing for multi-vector retrieval? Instead of random hyperplanes, partitioning based on the actual covariance structure of token embeddings—something like k-means-based LSH—would be much more robust to anisotropic data. PLAID’s centroid codebook is already a step in this direction; the question is whether a similar idea can be incorporated into MUVERA’s FDE construction.

What’s the right similarity function for visual retrieval? ColPali uses MaxSim because ColBERT used MaxSim—but is that right for image patches? A patch that depicts a chart legend might be uniformly relevant to many query tokens, which is a more SumSim-like interaction. Benchmarking different similarity functions on visual retrieval tasks seems underexplored.

How does retrieval geometry change with model scale? Larger VLMs (ColQwen, ColGemma) likely produce patch embeddings with different geometric properties than PaliGemma-based ColPali. Understanding how geometry scales with model capacity could inform both model design and infrastructure choices.

Can training objectives directly optimize for indexing efficiency? The variance regularization idea above is speculative, but the general question—can we train embeddings that are simultaneously high-quality for retrieval and favorable for ANN indexing—seems worth pursuing as a joint optimization problem.

These are the questions I’m watching closely as multi-vector retrieval continues to mature. The one thing I’m confident of: the conversation will be richer when it’s grounded in geometry and scoring functions, not just benchmark numbers on MS MARCO.

As always, the analysis here represents my current thinking based on the public literature and my own experimentation. If you have empirical results that confirm or challenge any of these geometric intuitions, I’d love to hear about it.

Previous in the series: When the Database Becomes the Bottleneck: The Economics of Vector Retrieval and the Technologies That Change the Math

Previous: The Reality of Vector Quantization: A Deep Dive into Product Quantization Trade-offs

References

1. Santhanam, K., Khattab, O., Potts, C., & Zaharia, M. (2022). PLAID: An Efficient Engine for Late Interaction Retrieval. Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM ‘22). arXiv:2205.09707

2. Dhulipala, L., Jayaram, R., Knittel, A., Liu, J., & Mirrokni, V. (2024). MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodings. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). arXiv:2405.19504

3. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2021). ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv:2112.01488.

4. Faysse, M., et al. (2024). ColPali: Efficient Document Retrieval with Vision Language Models. arXiv:2407.01449.

5. van den Oord, A., Li, Y., & Vinyals, O. (2018). Representation Learning with Contrastive Predictive Coding. arXiv:1807.03748.

6. Wang, F., & Liu, H. (2021). Understanding the Behaviour of Contrastive Loss. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2495–2504.

7. Rusak, E., et al. (2024). AnInfoNCE: Anisotropic Contrastive Learning. ICLR Workshop on High-dimensional Learning Dynamics.

8. Charikar, M. S. (2002). Similarity Estimation Techniques from Rounding Algorithms. Proceedings of the Thirty-Fourth Annual ACM Symposium on Theory of Computing, 380–388. (SimHash / LSH for cosine similarity)

9. Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4), 824–836.

10. Mathisen, J., et al. (2024). Scaling ColPali to Billions of PDFs with Vespa. Vespa Blog. blog.vespa.ai/scaling-colpali-to-billions

11. Herman, S. (2025). When the Database Becomes the Bottleneck: The Economics of Vector Retrieval and the Technologies That Change the Math. Blog post

12. Herman, S. (2025). Teaching Embedding Models New Words: A Deep Dive into Domain Adaptation. Blog post

13. Herman, S. (2025). The Reality of Vector Quantization: A Deep Dive into Product Quantization Trade-offs. Blog post

In my previous post on domain-adapted neural search, I argued that embedding quality is the most underleveraged factor in retrieval systems—easy to skip in the rush to get a pipeline running, but often the single highest-impact investment you can make. Domain adaptation, vocabulary extension, and careful fine-tuning can yield dramatic relevance improvements that no amount of index tuning can replicate.

That remains true. But now I want to complete the picture—because both halves matter, and the question of which half matters more depends entirely on scale.

At small corpus sizes, embedding quality dominates the cost-quality equation. As the corpus grows, storage and retrieval costs climb linearly while inference stays nearly flat—crossing over around 50–100 million vectors. We’ll derive these curves in detail below.

Once you’ve invested in getting your embeddings right, the database technology becomes absolutely critical. And not in the way most people think.

The database doesn’t just store and retrieve vectors. At scale, it determines whether your retrieval improvements are economically feasible to operate. It determines whether a research result translates into a production system that a business can actually run.

But the relationship runs deeper than a one-way pipeline from “embedding model” to “database.” The influence is bidirectional. When ColBERT’s multi-vector representations proved too expensive to index and retrieve at scale, the community didn’t just wait for better hardware—it pushed back to the embedding design itself. SPLADE emerged as a direct response: a learned sparse model engineered to be compatible with existing inverted index infrastructure, deliberately trading some of ColBERT’s expressiveness for the ability to leverage the most battle-tested indexing technology available. And then, when new indexing algorithms (PLAID, MUVERA) made multi-vector retrieval economically viable again, the pendulum swung back—enabling richer representations like ColPali and ColQwen that wouldn’t have been pursued without confidence that the infrastructure could support them.

This bidirectional feedback loop—where embedding models shape what database technology needs to exist, and database economics shape what embedding models get designed—is one of the central themes of this post. It’s the reason the story of retrieval isn’t a clean linear progression, but a conversation between two evolving fields.

In this post, I want to build an economic framework for understanding when and why this shift happens, and then explore two technologies—PLAID and MUVERA—that are reshaping what’s possible in the world of multi-vector retrieval.

The Economic Pivot Point

When Inference Dominates

At small to moderate scale, the dominant cost in a neural search system is embedding inference—the GPU compute required to encode queries and documents into vectors. Let’s build some intuition for why, and ground our cost estimates in the specifics of what drives inference pricing.

What Drives Embedding Inference Cost?

Embedding inference cost is fundamentally a function of model size (parameter count), hardware utilization, and throughput. These factors interact in ways that are worth understanding before we plug numbers into a spreadsheet.

Model parameters and their cost implications: Embedding models span a wide range of sizes. At the compact end, all-MiniLM-L6-v2 is a 6-layer transformer with ~22 million parameters producing 384-dimensional embeddings—small enough to run efficiently on a single CPU. Mid-range models like bge-base-en-v1.5 and GTE-base are built on BERT-base architectures (~110M parameters, 768 dimensions), while bge-large-en-v1.5 and GTE-large use BERT-large backbones (~335M parameters, 1024 dimensions). The newest generation of LLM-based embedding models—e5-mistral-7b-instruct, gte-qwen2-7b—scale up to 7 billion parameters but produce higher-quality embeddings, particularly for retrieval tasks.

The relationship between parameter count and cost is roughly linear for transformer-based models: more parameters means more floating-point operations per token, which translates directly to GPU-seconds per encoded document. A 7B parameter embedding model requires approximately 30–60× more compute per token than a 110M parameter model, but also tends to deliver meaningfully better retrieval quality.

Hardware factors: Inference cost also depends on the GPU (or CPU) hardware, its memory bandwidth, and how efficiently the deployment batches requests. An NVIDIA H100 can process embedding batches far faster than an A10G, but also costs proportionally more per hour. The key metric is throughput (tokens/second/dollar)—and here, newer hardware generally wins. On an H100 (with FP16 inference and dynamic batching), a 110M parameter model like bge-base-en-v1.5 can encode upwards of 10,000 documents per second, while a 7B parameter model might manage 200–500 documents per second on the same hardware.

Managed API vs. self-hosted economics: Cloud API providers (OpenAI, Google, Cohere) fold all of this into a per-token price that amortizes hardware, batching efficiency, and operational overhead. Self-hosted deployments on dedicated GPUs can achieve significantly lower per-token costs at high utilization, but require engineering investment in serving infrastructure (e.g., HuggingFace Text Embeddings Inference, vLLM, or ONNX Runtime).

The following chart visualizes the relationship between model size, throughput, and self-hosted cost:

Figure 1: Larger models produce better embeddings but at sharply lower throughput and higher cost. A 7B parameter model is ~100× slower than a 22M parameter model—but may deliver meaningfully better retrieval quality. The right choice depends on your quality requirements and scale.

With that context, here are representative 2025 API pricing benchmarks:

Note: OpenAI and Google do not disclose parameter counts for their embedding models, but pricing tiers strongly suggest the “small” variants use compact architectures (likely in the 100–300M range) while “large” variants use substantially bigger backbones. The pricing difference—6.5× between OpenAI’s small and large—is consistent with the compute scaling you’d expect from moving to a model with roughly 5–10× more parameters.

For a corpus of 1 million documents averaging 200 tokens each:

At this scale, the numbers are trivially small across the board. The inference cost is essentially a rounding error. The storage cost is pocket change. Everything fits in RAM on a modest instance. The embedding model quality is the only thing that matters.

This is the regime where my previous blog post’s thesis holds with full force: invest everything in getting the embeddings right, because the infrastructure is cheap.

When Storage and Retrieval Dominate

Now let’s scale up. Something interesting happens as the dataset grows but query volume remains moderate—a pattern common in enterprise search, legal discovery, patent databases, scientific literature, and compliance systems where large corpora serve a relatively fixed user base.

Before we look at the numbers, let’s ground the cost components in actual cloud infrastructure pricing so the estimates that follow aren’t black boxes.

Grounding the Infrastructure Costs

Storage costs are relatively straightforward. Cloud block storage (AWS EBS gp3, GCP Persistent Disk) runs $0.08–$0.10/GB-month. Object storage (S3, GCS) is cheaper at ~$0.023/GB-month but introduces latency that makes it unsuitable for active vector indices. For OpenSearch Serverless, managed storage on S3 is billed at $0.024/GB-month. The estimates below use $0.08/GB-month as a representative mid-point for block storage suitable for active vector workloads.

RAM and compute costs are where things get interesting—and expensive. Vector similarity search with HNSW requires the index to reside in memory. As I discussed in Part 1 of this series on GPU acceleration for vector search, the raw vector data for an HNSW index already scales linearly ($N \times D \times B$ bytes), but the graph structure itself adds 20–40% overhead on top. For 768-dimensional Float32 vectors, each node in the HNSW graph requires the 3,072-byte vector plus ~160 bytes for neighbor links (with typical M=16–32), bringing total in-memory footprint to roughly 1.3–1.5× the raw vector data size.

On AWS, memory-optimized instances commonly used for vector workloads price out as follows:

Prices shown for us-east-1; reserved instances reduce costs by 30–40%. Similar pricing exists on GCP (n2-highmem series) and Azure (E-series v5).

For managed services like OpenSearch Serverless, compute is measured in OCUs (OpenSearch Compute Units). Each OCU provides 6 GB of RAM and corresponding vCPU at $0.24/OCU-hour ($175/OCU-month). Vector search collections require their own dedicated OCU pool, and the minimum deployment is 2 OCUs ($350/month) for a dev/test configuration or 4 OCUs ($700/month) for production with HA.

ANN retrieval compute is the subtlest cost to estimate. The compute required per query depends on the HNSW parameters (M, efSearch), vector dimensionality, and the number of distance calculations performed during graph traversal. Based on benchmarks from the OpenSearch HNSW hyperparameter guide and consistent with my own findings in Part 1 of this series, a single HNSW query on 768-dimensional vectors with efSearch=64 typically requires 500–2,000 distance computations. On modern CPUs with SIMD acceleration (as we explored with AVX-512 and NEON in Part 1), each distance computation takes on the order of 0.1–0.5 μs, putting single-query latency in the 1–5 ms range for million-scale indices and 5–20 ms for billion-scale.

The cost implication: for moderate query volumes (1K–10K queries/day), a single memory-optimized instance has more than enough compute headroom—the bottleneck is purely RAM capacity. But at high query volumes (100K+ queries/day) or with larger vectors, you need to distribute across multiple nodes for both memory capacity and query throughput, and the compute cost becomes non-trivial.

With all that grounding in place, here are the estimates:

The Economics at 100 Million Vectors (768D Dense, Float32)

The Economics at 1 Billion Vectors (768D Dense, Float32)

Note: At billion scale, most deployments use a combination of product quantization (reducing in-memory footprint by 4–32×, as covered in Part 2 on PQ trade-offs) and disk-backed approaches like DiskANN to bring RAM costs down significantly. The numbers above represent the unoptimized upper bound.

The following chart illustrates this dynamic on a log-log scale. The inference cost curve (blue) remains nearly flat as the corpus grows—it’s bounded by query volume, not corpus size. The storage and retrieval cost curve (red) climbs linearly with corpus size. At the pivot zone (roughly 50–100 million vectors), the curves cross.

Figure 2: Cost crossover on a log-log scale. Inference cost scales with query volume (nearly flat), while storage and retrieval costs scale linearly with corpus size. The crossover at ~50–100M vectors marks the transition from an “embedding-quality-dominated” to a “database-technology-dominated” cost regime.

The pattern is unmistakable. Query inference remains essentially flat—it scales with query volume, which is bounded by your user base. But storage and retrieval costs scale linearly with corpus size. At some point between 10 million and 100 million vectors, the cost curve crosses: storage-at-rest and online retrieval become the dominant expense.

The Pivot Point: When corpus growth outpaces query volume growth—which is the case for the vast majority of enterprise search applications—the database technology becomes the critical cost driver, typically somewhere around 50–100 million vectors.

This is the moment where the index technology transitions from “implementation detail” to “strategic decision.” And this is where the choice between dense, sparse, and late interaction embeddings stops being a purely academic question and becomes a financial one.

The Cost of Representation: Dense vs. Late Interaction vs. Sparse

Now that we understand when the database becomes critical, let’s compare the three major embedding paradigms through an economic lens. But before diving into the numbers, it’s worth framing what we’re about to see—because the order in which these representations were developed tells a story about that bidirectional feedback loop.

Dense bi-encoders came first: simple, efficient, one vector per document. They worked well with nascent ANN indexing technology and scaled reasonably. Then came ColBERT, offering richer per-token representations with measurably better retrieval quality. The theory was compelling—but as we’ll see in the numbers below, ColBERT’s downstream storage and retrieval costs hit the pivot point almost immediately, making it impractical at any meaningful scale. The infrastructure simply couldn’t support the representation.

That economic reality didn’t just stay in the infrastructure layer—it fed back into embedding research. SPLADE was, in many ways, a direct answer to this feedback: “If the best representations can’t be efficiently indexed, can we design representations that deliver most of the quality while fitting the indexing technology we already have?” The result was a learned sparse model that leverages inverted indices—trading some theoretical expressiveness for massive economic viability.

Understanding this history matters because the numbers below aren’t just a static comparison. They explain why the field evolved the way it did, and they set the stage for understanding why PLAID and MUVERA matter: they’re the infrastructure catching up, finally making richer representations viable again.

Dense Embeddings: The Baseline

Let’s follow that chronological thread and start with the simplest representation. A dense bi-encoder produces one vector per document. For a model with 768 dimensions at Float32 precision:

Dense embeddings enjoy the most mature indexing ecosystem. Every major vector database supports them natively. ANN algorithms are highly optimized for single-vector search, and as I covered in Part 2 of this series, product quantization can compress these vectors significantly (32:1 at the usable end of the spectrum) to bring costs down further.

Late Interaction Embeddings: The Quality Premium

Dense embeddings set a solid baseline—but collapsing an entire document into a single vector inevitably loses information. ColBERT and its successors (ColBERTv2, ColPali, ColQwen) address this by producing one vector per token in a document, preserving fine-grained token-level interactions that a single pooled vector necessarily discards. For ColBERTv2 with 128-dimensional embeddings and an average document length of 100 tokens:

Even with ColBERTv2’s aggressive 2-bit residual compression (we’ll examine how this works in detail when we discuss PLAID later), the storage footprint remains comparable to uncompressed dense vectors. But the real cost isn’t just storage—it’s the retrieval compute. The MaxSim operation requires comparing every query token against every document token for each candidate, making brute-force retrieval approximately 100× more expensive than dense single-vector search (assuming ~32 query tokens × ~100 document tokens vs. a single dot product).

For a long time, this cost differential made late interaction a theoretical curiosity—wonderful on benchmarks, impractical in production. The retrieval quality improvements (often 5–15% nDCG improvement over dense models on standard benchmarks) simply weren’t worth the 16–50× increase in storage and 100× increase in retrieval compute at scale.

This is our first concrete example of the database feeding back to the embedding model. ColBERT was designed to solve real problems with dense retrieval: better interpretability (you can trace which tokens matched), finer-grained relevance signals, and measurably higher quality. On its own merits, it was a clear advance. But when these representations met production-scale infrastructure, the economics were prohibitive. The downstream cost of storing and retrieving multi-vector embeddings didn’t just make ColBERT expensive—it made researchers question whether the approach was viable at all, and directly motivated the search for alternatives that could deliver similar quality within the constraints of existing infrastructure.

The Late Interaction Paradox: Late interaction models consistently deliver the best retrieval quality, but for years, there was no efficient index that could capitalize on those theoretical benefits at production scale. The database pushed back—and embedding researchers listened.

Sparse Embeddings (SPLADE): The Efficiency Champion

And that listening produced our third paradigm. SPLADE is where the feedback loop from database to embedding model becomes most visible. Learned sparse models didn’t emerge in a vacuum—they were a direct response to the economic impasse created by late interaction. When ColBERT’s multi-vector representations proved too expensive to index and retrieve at production scale, and even dense ANN indices carried significant storage costs that ballooned with corpus size, researchers asked a different question: instead of building new indexing technology for richer representations, can we design representations that work with the indexing technology we already have?

SPLADE and similar learned sparse models are the answer. Instead of dense vectors, they produce sparse representations in vocabulary space (~30,000 dimensions for BERT-based models), where each non-zero dimension corresponds to a specific vocabulary term. This is a deliberate compromise on the expressiveness of ColBERT’s per-token embeddings—but it’s a compromise made with clear economic intent. By mapping into vocabulary space, SPLADE embeddings are natively compatible with inverted indices, the most mature and cost-efficient indexing technology available.

Here’s what makes SPLADE particularly interesting from an economic standpoint: it delivers retrieval quality approaching late interaction models (within 2–5% on many benchmarks) while leveraging the most battle-tested indexing technology in existence—inverted indices. No specialized infrastructure needed. No custom ANN algorithms. Your existing Elasticsearch or OpenSearch cluster can serve SPLADE embeddings with minimal modification.

And recall from my previous blog that in our domain adaptation experiment, we compressed the effective SPLADE dimensionality to only 11 non-zero terms for domain-specific queries. That’s another order of magnitude in storage and compute reduction.

The Cost Comparison at Scale

Let’s put this all together for 1 billion documents:

The Insight: SPLADE delivers 80–90% of late interaction quality at a fraction of the cost—often cheaper than even dense vectors. For text-only retrieval with well-adapted models, SPLADE is remarkably hard to beat on a cost/quality basis.

But the table alone doesn’t capture how dramatically different these cost trajectories are as corpus size grows. The following chart shows monthly infrastructure cost (storage + retrieval compute) for each representation type on a log-log scale, with a horizontal reference line representing a typical inference budget (~$100/month). The point where each curve crosses that line is its pivot point—where the database technology overtakes embedding quality as the dominant cost factor:

Figure 3: Infrastructure cost vs. corpus size for each embedding representation. Vanilla late interaction hits its pivot point at just ~2M documents—meaning infrastructure dominates cost almost immediately. Dense PQ reaches it around ~50M. Domain-optimized SPLADE doesn’t cross until ~350M documents, giving you an enormous range where embedding quality investments remain the better use of your budget.

The spread is striking. Late interaction’s 100× retrieval cost multiplier means it enters “infrastructure-dominated” territory at corpus sizes where other representations are still comfortably in the “embedding-quality-dominated” phase. This is precisely why late interaction was considered impractical for so long—and why PLAID and MUVERA (which we’ll explore next) represent such a significant shift in the economics.

The VLM Plot Twist: Why Late Interaction Is Back

If the story ended here, the conclusion would be straightforward: invest in SPLADE, adapt it to your domain, and enjoy near-SOTA quality at rock-bottom infrastructure costs.

But the story doesn’t end here.

Vision-Language Models (VLMs) and multi-modal encoders are changing the game. Models like ColPali and ColQwen use late interaction architectures to encode images of documents—PDF pages, slides, diagrams, scanned forms—into multi-vector representations. Each image patch becomes a token embedding, preserving spatial and visual information that simply cannot be captured by text-only models.

This is transformative for industries drowning in visual documents: financial reports with embedded charts, medical records with imaging, legal contracts with signatures and stamps, engineering specs with diagrams.

The problem is that there’s no clean way to translate rich visual representations into sparse embeddings. You could run OCR, generate text descriptions, and encode those with SPLADE—but you’d lose the spatial relationships, the chart semantics, the diagram context. The latent visual information in those patch embeddings is precisely what makes multi-modal late interaction so powerful.

Late interaction is back in play. Not because the text retrieval community suddenly discovered it’s better (they always knew), but because multi-modal applications require multi-vector representations, and the quality gap between “real” multi-vector similarity and any text-based approximation is too large to ignore.

Which brings us to the crux of this post: if late interaction is necessary, how do we make it economically viable?

The answer comes from two technologies that attack different parts of the cost equation: PLAID (storage and candidate generation) and MUVERA (retrieval compute).

PLAID: Making Late Interaction Storage-Efficient

While SPLADE represented the field’s adaptation away from multi-vector representations, the ColBERT authors took a different path. Rather than abandoning the expressiveness of late interaction, they turned the feedback loop into a research agenda: if the database infrastructure is the bottleneck, fix the database infrastructure.

This response was systematic. ColBERTv2 (2021) addressed the storage problem—introducing 2-bit residual compression that reduced per-document footprint from ~51 KB to ~2 KB, shrinking the gap from 16× to roughly comparable with uncompressed dense vectors. But storage was only half the equation. The retrieval compute—the cost of actually searching those multi-vector representations—remained the dominant expense.

PLAID (2022) followed as the direct answer to the retrieval cost problem. It represents a fascinating example of how algorithmic advances in ANN indexing technology can fundamentally reshape what’s economically viable. Where SPLADE asked “can we change the embedding to fit existing infrastructure?”, PLAID asked “can we build new infrastructure that makes the best embeddings affordable?” As we’ll see, the answer turned out to be a qualified but compelling yes—and in doing so, PLAID made late interaction vectors potentially economically viable for the first time.

Before we can understand PLAID’s innovation, we need to understand what ColBERTv2 gives us to work with—because PLAID’s four-stage pipeline is built directly on top of ColBERTv2’s internal representation structure.

ColBERTv2’s Residual Compression: The Foundation

The original ColBERT stored every token embedding as a full 128-dimensional Float32 vector—512 bytes per token. For a 100-token document, that’s 51,200 bytes, and at billion scale, roughly 50 TB of storage. ColBERTv2’s central contribution was recognizing that these token embeddings are highly redundant: tokens with similar semantic roles (e.g., all instances of “financial” across the corpus) cluster tightly in embedding space. If you can identify those clusters, you only need to store the difference from the cluster center—a much smaller quantity.

The encoding algorithm works in four steps:

1. Learn a centroid codebook. During offline indexing, ColBERTv2 runs k-means clustering over all token embeddings in the corpus to produce $C = 2^{16} = 65{,}536$ centroids. Each centroid is a 128-dimensional vector representing a cluster of semantically similar tokens. The number 65,536 is chosen so that each centroid can be addressed by a 16-bit (2-byte) integer.

2. Assign each token to its nearest centroid. For every token embedding $t$ in every document, find the closest centroid:

This centroid ID is stored as a 2-byte integer.

3. Compute the residual. The residual is the difference between the original embedding and its assigned centroid:

Because the centroid is already a good approximation of the token, the residual values are small—they cluster near zero. This is the key property that makes them highly compressible.

4. Quantize the residual to 2 bits per dimension. ColBERTv2 learns per-centroid quantization boundaries from the training data. Each of the 128 residual dimensions is mapped to one of 4 buckets (encoded as 2 bits): large negative, small negative, small positive, large positive. This yields $128 \times 2 = 256$ bits $= 32$ bytes of quantized residual per token, plus a small amount of metadata.

The following diagram traces a single token through this pipeline:

Figure 4: ColBERTv2 compresses each token from 512 bytes (Float32) to ~20 bytes (centroid ID + quantized residual)—a 25.6× reduction. The key insight is that the residual values cluster near zero because the centroid is already close to the original embedding, making aggressive quantization feasible with minimal information loss.

Numerical Example: Encoding a Single Token

Let’s trace through concrete numbers. Suppose a token embedding for the word “amortization” in a financial document is:

Step 1–2: K-means assigns this token to centroid $c_{17425}$, a cluster center representing financial terminology. The centroid vector is:

Step 3: The residual is:

Notice how much smaller these values are compared to the original embedding. The centroid captured the “financial terminology” signal; the residual captures the fine-grained distinction of “amortization” versus other financial terms.

Step 4: Each residual dimension is quantized. For centroid 17425, suppose the learned bucket boundaries are $[-0.06, 0, +0.06]$. Then:

The final stored representation for this token: centroid ID 17425 (2 bytes) + 256-bit quantized residual (~18 bytes with packing and metadata) = ~20 bytes total.

For a 100-token document, this yields ~2,000 bytes—compared to 51,200 bytes for the original Float32 representation. Across 1 billion documents, this reduces storage from ~50 TB to ~2 TB.

Decoding: Reconstructing Approximate Embeddings

To compute exact MaxSim at scoring time, ColBERTv2 reconstructs an approximate embedding by reversing the process:

where $r_q$ is the quantized residual and dequantize maps each 2-bit code back to its bucket’s representative value. The reconstructed $\hat{t}$ is close to the original $t$—typically within a cosine similarity of 0.95–0.99 of the original. This small error is acceptable because the final MaxSim scoring only needs to rank the top few candidates correctly, not reproduce exact scores.

The Three Representation Tiers

What makes ColBERTv2’s design particularly elegant—and what PLAID exploits—is that it naturally creates three tiers of representation with very different cost profiles:

Figure 5: ColBERTv2 stores each document at two tiers: centroid IDs (Tier 1, in RAM) and centroid ID + quantized residual (Tier 2, on disk). Tier 3 is reconstructed on-the-fly only when needed. PLAID’s insight is that Tier 1 alone—just the centroid IDs—carries enough signal to eliminate 99%+ of the corpus before touching the more expensive tiers.

The cost implications of this tiered structure are dramatic:

This is the foundation that PLAID builds on. The question PLAID answers is: can we use Tier 1 (the cheapest representation) to do most of the work, only falling back to Tier 2 and Tier 3 for the small number of documents that survive initial filtering?

The Core Problem PLAID Solves

Recall the fundamental challenge of ColBERT retrieval. You have a query with ~32 token embeddings and a corpus of billions of document token embeddings. For each query, you need to compute MaxSim—the sum of maximum cosine similarities between each query token and all document tokens:

Vanilla ColBERTv2 handles this in two stages: first, it uses an inverted list of centroids to generate candidate documents, then it decompresses the full residual-encoded token embeddings for each candidate and computes exact MaxSim scores. Let’s unpack how this works.

During indexing, ColBERTv2 builds an inverted index from centroid IDs to documents. For each of the 65,536 centroids in the codebook, the index stores a posting list: the set of (document, token position) pairs where a token was assigned to that centroid. This is conceptually identical to a traditional search engine’s inverted index—except instead of mapping words to documents, it maps embedding cluster IDs to documents.

At query time, each of the ~32 query token embeddings is compared against the codebook to find its nearest centroids. The inverted lists for those centroids are retrieved, and the union across all query tokens produces the initial candidate set—typically around 50,000 documents for a corpus of 140 million. This first stage is fast because it operates entirely on Tier 1 data in RAM: just centroid IDs and posting lists.

The expensive part is what comes next. For every candidate document, vanilla ColBERTv2 reads the full compressed representation from disk (Tier 2), reconstructs approximate token embeddings (Tier 3), and computes exact MaxSim across all query-document token pairs. At 50,000 candidates, that’s 100 MB of disk reads and 160 million similarity computations per query (50,000 documents × 32 query tokens × 100 document tokens per document).

Figure 6: The two-stage vanilla ColBERTv2 retrieval process. The centroid codebook and inverted lists (Tier 1) live in RAM for fast candidate generation. But all ~50,000 candidates must then be decompressed from disk and scored with exact MaxSim—the bottleneck that PLAID addresses by inserting additional centroid-only filtering stages before any disk access.

The problem is that the candidate generation stage is too coarse—it retrieves far more documents than necessary—and the decompression stage is too expensive to apply to all of them. Even with ColBERTv2’s residual compression, retrieving and scoring thousands of candidates requires touching enormous amounts of data.

PLAID’s Insight: Bags of Centroids

PLAID (Performance-optimized Late Interaction Driver) introduces a beautifully simple observation: you don’t need the full token embeddings to eliminate most of the corpus. In terms of the tier framework above, PLAID recognizes that Tier 1 alone—just the centroid IDs, at 200 bytes per document—carries enough signal to prune the vast majority of non-relevant documents.

Here’s the key intuition. If a document’s tokens are all assigned to centroids that are distant from all query token embeddings, that document is almost certainly irrelevant. We don’t need to decompress a single residual to figure this out—we can make that determination using only the centroid assignments. PLAID builds an inverted index from centroid ID to document list, enabling sub-millisecond lookup of which documents contain tokens assigned to any given centroid.

The following diagram puts this in contrast with vanilla ColBERTv2. Where ColBERTv2 jumps directly from candidate generation to full decompression of all ~50,000 candidates, PLAID inserts two additional filtering stages that operate entirely on Tier 1 data—eliminating 98% of candidates before touching disk:

Figure 7: Side-by-side comparison. Vanilla ColBERTv2 (left) decompresses and scores all ~50,000 candidates at full cost—100 MB of disk reads and 160M similarity computations per query (50K docs × 32 query tokens × 100 doc tokens). PLAID (right) inserts two Tier-1-only stages (centroid pruning and centroid interaction scoring) that reduce the candidate set to ~1,000 before any disk access, cutting I/O by 50× and latency by 45× on CPU—with zero quality loss.

Let’s now walk through each of PLAID’s four stages in detail.

The PLAID Pipeline

PLAID implements a four-stage scoring pipeline, each stage progressively refining the candidate set:

Stage 1: Candidate Generation via Centroid Lookup

For each query token embedding, PLAID looks up the nearest centroids in the codebook. It then retrieves all documents that have any token assigned to those centroids. This produces a large initial candidate set—potentially tens of thousands of documents—but it’s extremely fast because it operates entirely on Tier 1 data: an inverted index lookup on 2-byte centroid IDs, all resident in RAM.

Stage 2: Centroid Pruning

Not all centroids in a document contribute meaningfully to the MaxSim score. PLAID computes the similarity between each centroid in the candidate documents and the query token embeddings. Centroids with low similarity to all query tokens are pruned—effectively “sparsifying” the document representation.

This is mathematically justified because the MaxSim operation takes the maximum similarity for each query token. If a document centroid isn’t the best match for any query token, it cannot contribute to the final score, so we can safely ignore it.

Figure 8a: Centroid pruning applied to candidate document d₇. Each of d₇’s 100 centroids is tested against all query tokens; the maximum similarity to any query token determines whether it survives. Centroids like c₄₁₂₀₆ (max_sim = 0.15) fall below the threshold τ = 0.50 and are discarded. The result is a sparsified representation—only ~25–30 centroids survive from the original 100, reducing the per-document footprint from 200 bytes to ~50 bytes. All similarities come from the centroid codebook already in RAM; no disk access is needed.

Stage 3: Centroid Interaction Scoring

Using only the surviving (non-pruned) centroids, PLAID computes an approximate MaxSim score. Since the centroids come from a fixed codebook, the query-to-centroid distances can be precomputed once and reused across all documents—the same memoization trick we discussed in the context of Product Quantization, applied here to centroid interactions.

This stage dramatically reduces the candidate set. Documents that score poorly using centroid-only similarity are eliminated without ever touching their full representations.

Figure 8b: Centroid interaction scoring for the same query. Step A precomputes a 32 × 65,536 lookup table of query-token↔centroid similarities (~8 MB, computed once). Step B scores each candidate using only its surviving centroids—for d₇, the approximate MaxSim sums the best centroid match per query token (0.87 + 0.79 + 0.82 + …). Step C ranks all 50,000 candidates and keeps the top ~1,000. The 49,000 eliminated candidates never trigger a disk read. Total scoring time: ~6 ms.

Stage 4: Full Residual Decompression and Exact Scoring

Only for the small number of surviving candidates (typically a few hundred out of the original millions) does PLAID fall through to Tier 2 and Tier 3: reading the quantized residuals from disk, reconstructing approximate embeddings, and computing exact MaxSim. This is the expensive step, but it now applies to a tiny fraction of the corpus.

The following diagram traces a concrete query through all four stages:

Figure 9: A 32-token query against 140M passages. Each stage dramatically narrows the candidate set while using progressively more expensive scoring. The total data decompressed drops from 280 GB (brute force) to just 2 MB—a 140,000× reduction.

Worked Example: A Single Query Through PLAID

Let’s trace the numbers explicitly. Suppose we have a 32-token query searching 140 million passages (MS MARCO scale), where each passage averages 100 tokens encoded with ColBERTv2 (128-dim embeddings, 2-bit residual compression):

The critical insight is the data amplification ratio: Stage 1 operates on the cheapest possible representation (centroid IDs) against the largest candidate set, while Stage 4 operates on the most expensive representation (full residuals) against the smallest candidate set. This is precisely the right ordering for cost efficiency.

Why This Works: The Mathematics of Centroid Sufficiency

The key theoretical insight behind PLAID is that centroid-only retrieval exhibits high recall relative to full ColBERT scoring. The paper demonstrates that if you retrieve 10× k documents using only centroid similarity, those documents contain 99%+ of the true top-k results from the full pipeline.

This follows directly from the residual decomposition we established above. Recall that for a token embedding $t$ assigned to centroid $c$ with residual $r = t - c$:

Because we designed the codebook to minimize residual magnitude ($|r|$ is small by construction), the centroid similarity $\cos(q, c)$ is a strong proxy for the true token similarity $\cos(q, t)$. When we’re only trying to rank documents (not compute exact scores), the centroid similarity is usually sufficient to identify the right candidates—and we defer the residual correction to Stage 4 for exact scoring.

PLAID Performance Results

The results are striking. On MS MARCO passage retrieval at 140 million passages:

PLAID achieves these speedups without any degradation in retrieval quality. The MRR@10 is identical to vanilla ColBERTv2 because the final stage still computes exact MaxSim—PLAID only accelerates the candidate filtering, not the final scoring.

The side-by-side comparison makes the impact concrete:

Figure 10: Vanilla ColBERTv2 vs. PLAID at 140M passages. PLAID reduces the in-memory working set by 10×, GPU latency by 7×, CPU latency by 45×, and per-query data decompression by 50×—all with zero quality loss.

What PLAID Saves: PLAID primarily reduces retrieval latency and storage I/O by keeping the working set (centroid IDs) small enough to fit in memory while deferring expensive decompression to only the most promising candidates. It transforms ColBERT from a system that requires touching terabytes of data per query into one that touches megabytes.

MUVERA: Reducing Multi-Vector Search to Single-Vector MIPS

The Problem PLAID Doesn’t Solve

PLAID is a remarkable engineering achievement, but it has an architectural limitation: it requires a custom retrieval pipeline. You can’t simply drop ColBERT embeddings into an off-the-shelf vector database and expect efficient retrieval. PLAID needs its own inverted index structure, its own centroid pruning logic, and its own multi-stage pipeline.

This matters because the single-vector MIPS (Maximum Inner Product Search) ecosystem has had decades of optimization. Algorithms like HNSW, IVF, and DiskANN are battle-tested, highly optimized, and available in every major vector database. If we could somehow reduce multi-vector retrieval to single-vector MIPS, we could leverage this entire ecosystem for free.

That’s exactly what MUVERA does.

MUVERA’s Key Idea: Fixed Dimensional Encodings

MUVERA (Multi-Vector Retrieval Algorithm) introduces a mathematical transformation that converts a variable-length set of vectors (a multi-vector embedding) into a single fixed-length vector called a Fixed Dimensional Encoding (FDE). The critical property of this transformation is that the dot product between two FDEs approximates the Chamfer similarity between the original multi-vector sets.

Let me unpack that because it’s a beautiful piece of mathematics.

Chamfer Similarity (the scoring function used by ColBERT and similar models) between a query set $Q = {q_1, …, q_m}$ and a document set $P = {p_1, …, p_n}$ is:

This is a non-linear, asymmetric function—the max operation prevents it from decomposing into a simple linear algebra operation. You can’t just concatenate or average the vectors and get the right answer.

MUVERA’s insight is to use randomized space partitioning to convert this non-linear problem into a linear one. Here’s how.

The FDE Construction Algorithm

Step 1: Partition the Embedding Space

MUVERA uses SimHash (a form of locality-sensitive hashing) to partition $\mathbb{R}^d$ into $K$ regions. The critical property of SimHash is that vectors that are close in the original space are likely to land in the same partition. Each partition roughly corresponds to a “neighborhood” of semantically similar token embeddings.

Step 2: Assign Vectors to Partitions

For each vector in a multi-vector set, MUVERA determines which partition it falls into. Vectors from the same semantic neighborhood (e.g., all token embeddings related to “financial analysis”) will tend to cluster in the same partitions.

Step 3: Aggregate Within Partitions

Within each partition, the vectors are aggregated—typically by summing and projecting to a lower dimension. This produces a fixed-size “summary” for each partition.

Step 4: Concatenate Partition Summaries

The per-partition summaries are concatenated into a single fixed-dimensional vector. The total FDE dimension is $r \times K \times d_{\text{proj}}$, where $r$ is the number of independent repetitions (for concentration of measure), $K$ is the number of partitions, and $d_{\text{proj}}$ is the projection dimension per partition.

Why This Preserves Similarity

The key mathematical insight is that if query token $q_i$ and its best-matching document token $p_{j^}$ are in the *same partition (which SimHash ensures with high probability for similar vectors), then the dot product of their respective partition summaries will capture that match. By repeating this process $r$ times with independent random partitions, MUVERA achieves concentration—the FDE dot product converges to a good approximation of the true Chamfer similarity.

Formally, MUVERA proves:

This is the first algorithm providing a single-vector proxy for multi-vector similarity with theoretical guarantees. Prior approaches were heuristics with no approximation bounds.

Asymmetric Encoding: A Crucial Detail

MUVERA’s encoding is asymmetric—the query FDE and document FDE are constructed differently. This is essential because Chamfer similarity itself is asymmetric: it sums over query tokens, not document tokens. The query encoding emphasizes which partitions are “active” (contain query tokens), while the document encoding ensures all document tokens contribute to their respective partitions regardless of the query.

This asymmetry means document FDEs can be precomputed and indexed offline, while query FDEs are computed at query time—exactly the pattern needed for efficient retrieval.

MUVERA’s Retrieval Pipeline

Compared to PLAID’s four-stage pipeline, MUVERA’s is refreshingly simple:

1. Offline: Compute document FDEs from ColBERT token embeddings. Index using any standard MIPS solver (e.g., DiskANN, HNSW).
2. Online: Compute query FDE. Run single-vector ANN search to retrieve top candidates.
3. Re-rank: Score candidates using exact Chamfer similarity on the original multi-vector representations.

That’s it. No custom inverted indices. No centroid pruning. No multi-stage pipeline. Just standard single-vector ANN search with a re-ranking step.

MUVERA Performance Results

On the BEIR benchmark suite with ColBERTv2:

MUVERA achieves its candidates with 2–5× fewer retrieved candidates than PLAID’s heuristic-based approach, meaning less data touched and less re-ranking needed.

The Trade-offs

MUVERA is not without costs:

FDE Size: The FDE vectors are large. A typical configuration (e.g., $r=20$ repetitions, $K=64$ partitions, $d_{\text{proj}}=32$ projection dimensions) produces FDEs of dimension $20 \times 64 \times 32 = 40,960$. That’s significantly larger than a typical dense embedding. However, these vectors are amenable to standard quantization techniques (PQ, scalar quantization) since they’re single vectors in a standard MIPS setup.

Recall Trade-off: The FDE approximation introduces some recall loss compared to exact Chamfer search. In practice, this is mitigated by the re-ranking step—the FDE only needs to be good enough to put the right documents in the candidate set, not to produce exact scores.

No Free Lunch: The combination of FDE storage plus the original multi-vector storage for re-ranking means MUVERA’s total storage footprint can be larger than PLAID’s. The win is in compute—using optimized MIPS infrastructure instead of custom multi-stage pipelines.

PLAID vs. MUVERA: Complementary Approaches

It’s tempting to ask “which one is better?” but that’s the wrong question. PLAID and MUVERA address different aspects of the late interaction cost problem:

In practice, these approaches may even be complementary. A system could use MUVERA’s FDE for initial candidate generation via an off-the-shelf ANN index, then use PLAID’s centroid interaction for an efficient intermediate scoring stage before final re-ranking with exact Chamfer similarity. This three-tier approach would combine MUVERA’s infrastructure compatibility with PLAID’s efficient intermediate scoring.

Revisiting the Decision: From “No” to “Maybe”

Let’s return to our economic framework. Before PLAID and MUVERA, the late interaction cost equation looked like this at 1 billion documents:

Before: Late Interaction (Vanilla ColBERTv2)

vs. Dense retrieval at the same scale:

Dense Retrieval (768D, PQ-compressed)

Late interaction was 25–100× more expensive. For a 5–15% quality improvement, that’s almost never justified.

After: Late Interaction with PLAID

After: Late Interaction with MUVERA

The cost gap has shrunk from 25–100× to 3–5×. For applications where the quality improvement matters—legal discovery, medical literature search, multi-modal document retrieval—this is now firmly in the territory of “worthwhile trade-off.”

And for multi-modal applications (ColPali, ColQwen) where there is no sparse embedding alternative, PLAID and MUVERA don’t just improve the economics—they make multi-vector retrieval possible at scale for the first time.

The following chart puts all five approaches on the same log-scale axis, making the shift visible at a glance:

Figure 11: Monthly infrastructure cost at 1B documents for all five retrieval approaches. The cost gap between late interaction and dense baselines has shrunk from 25–100× (a hard “no”) to 3–5× (a viable trade-off). SPLADE remains the text-only champion, but for multi-modal retrieval, PLAID and MUVERA are the only path to production scale. All costs derived from the pricing framework established earlier: $2.28/GB-RAM/month (r6g on-demand) and $0.08/GB-storage/month (EBS gp3).

The Verdict: What was once a hard “no” to late interaction at production scale has become a soft “yes”—or at minimum, a much softer “no.” For text-only retrieval where SPLADE is viable, sparse embeddings remain the cost-performance champion. But for multi-modal retrieval, and for text applications where that extra 5–15% quality matters, PLAID and MUVERA have changed the math.

The Feedback Loop: A Theme Revisited

We’ve seen this dynamic play out at every stage of the analysis. Index technology doesn’t just enable embedding models—it actively shapes which ones get designed, funded, and deployed.

ColBERT demonstrated the quality potential of late interaction, but infrastructure costs pushed the field toward SPLADE—a compromise engineered to fit existing inverted indices. ColBERTv2 and PLAID then pushed back from the infrastructure side, making multi-vector retrieval affordable enough that researchers felt confident investing in richer representations again. ColPali and ColQwen exist partly because the community now believes that multi-vector retrieval at scale is solvable.

This bidirectional feedback loop—where index capabilities inform embedding design, and embedding requirements drive index innovation—is one of the most productive dynamics in information retrieval research right now. It’s why I believe the next few years will see rapid progress on both fronts simultaneously.

Practical Recommendations

For practitioners deciding how to architect their retrieval systems today:

If your corpus is < 50 million documents and text-only: Invest your effort in embedding quality. Use dense or SPLADE embeddings. The index is not your bottleneck. Read my previous post on domain adaptation.

If your corpus is 50M–1B documents and text-only: Seriously evaluate SPLADE with domain adaptation. You’ll get near-SOTA quality with inverted index infrastructure you probably already have. Dense with PQ compression (see Part 2 of this series) is also viable if you need pure dense similarity.

If you need multi-modal retrieval (documents with images, charts, PDFs): Late interaction (ColPali/ColQwen) is your best path to quality. Use MUVERA if you want to leverage existing vector database infrastructure. Use PLAID if you’re building dedicated retrieval infrastructure and want the most mature solution.

If you’re at >1B scale with any embedding type: You’re in custom infrastructure territory. Evaluate PLAID + MUVERA as a combined pipeline. The economics at this scale demand careful benchmarking on your specific workload.

Looking Forward

We’re living through a remarkable period in information retrieval. The convergence of multi-modal models, efficient indexing algorithms, and dropping hardware costs is making retrieval architectures possible that were purely theoretical just two years ago.

The next frontiers I’m watching closely:

• MUVERA + Product Quantization: Compressing the large FDE vectors with PQ could dramatically reduce MUVERA’s storage overhead.
• Hardware-aware multi-vector indexing: Modern GPUs and custom accelerators are particularly well-suited to the batched matrix operations that MaxSim requires. PLAID on GPU is already 7× faster—purpose-built hardware could push this further.
• End-to-end learned quantization for late interaction: Rather than applying PQ post-hoc, training the embedding model to produce quantization-friendly representations could improve the compression-quality trade-off.
• Hybrid sparse + late interaction systems: Using SPLADE for initial retrieval and ColBERT for re-ranking combines the efficiency of sparse search with the quality of late interaction.

As we continue building developing the new more powerful generation of models such as ColPali, these technologies make it possible and therefore represent the cutting edge of what’s possible in production vector search. The goal, as always, is finding the optimal balance between scale, speed, quality, and cost.

The database was never “just” the database. At scale, it’s the enabler that determines whether your AI can actually serve users—or remains a promising benchmark result gathering dust on a researcher’s laptop.

This analysis is based on publicly available research papers, benchmark results, and cloud pricing data as of early 2026. Actual costs vary significantly based on cloud provider, instance type, query patterns, and dataset characteristics. Always benchmark on your specific workload.

Previous in the series: The Reality of Vector Quantization: A Deep Dive into Product Quantization Trade-offs

Previous: Teaching Embedding Models New Words: A Deep Dive into Domain Adaptation

References

1. Santhanam, K., Khattab, O., Potts, C., & Zaharia, M. (2022). PLAID: An Efficient Engine for Late Interaction Retrieval. Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM ‘22). arXiv:2205.09707

2. Dhulipala, L., Jayaram, R., Knittel, A., Liu, J., & Mirrokni, V. (2024). MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodings. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). arXiv:2405.19504

3. Khattab, O. & Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.

4. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2021). ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv:2112.01488.

5. Faysse, M., Fernandes, H., Music, N., Music, E., Music, N. (2024). ColPali: Efficient Document Retrieval with Vision Language Models. arXiv:2407.01449.

6. Formal, T., Piwowarski, B., & Clinchant, S. (2021). SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. Proceedings of the 44th International ACM SIGIR Conference.

7. Jégou, H., Douze, M., & Schmid, C. (2011). Product Quantization for Nearest Neighbor Search. IEEE TPAMI, 33(1), 117-128.

8. Subramanya, S. J., et al. (2019). DiskANN: Fast Accurate Billion-Point Nearest Neighbor Search on a Single Node. NeurIPS 2019.

9. Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE TPAMI, 42(4), 824-836.

10. Herman, S. (2025). Teaching Embedding Models New Words: A Deep Dive into Domain Adaptation. Blog post.

11. Herman, S. (2025). The Reality of Vector Quantization: A Deep Dive into Product Quantization Trade-offs. Blog post.

Neural embedding models have revolutionized information retrieval, yet their practical deployment in enterprise environments remains challenging. While these models excel on benchmark datasets, they consistently underperform on domain-specific corpora containing specialized terminology, acronyms, and jargon absent from their pre-training vocabulary.

This investigation started with a simple question: Can I teach a pre-trained embedding model to recognize and semantically reason about entirely novel vocabulary?

Through controlled experimentation with the SPLADE architecture, I discovered that naive fine-tuning approaches fail to achieve meaningful domain adaptation. The root cause—insufficient MLM (Masked Language Modeling) logit activation for out-of-vocabulary terms—led me down a rabbit hole that ultimately produced a multi-stage adaptation pipeline that actually works.

What I’ll cover:

1. Why standard fine-tuning fails for domain vocabulary adaptation
2. A diagnostic framework for identifying vocabulary integration failures at the MLM logit level
3. A staged adaptation methodology combining vocabulary extension, domain warmup pre-training, dictionary-based pre-training, and multi-objective fine-tuning
4. How to achieve successful semantic query expansion to novel domain terminology

Introduction

The Domain Adaptation Challenge

At OpenSearchCon 2025 NA, I challenged a common misconception in the vector search space: that the index is the primary driver of good recall. Marketing teams and vendors love to focus on index architecture, but my research pointed to a different culprit. The main factor degrading recall and search relevancy in production isn’t the index—it’s the quality of the embeddings themselves.

To put this in perspective: a typical ANN index (HNSW, IVF, etc.) achieves 95-99% recall relative to exact k-NN search. That’s a 1-5% loss from approximate indexing. Meanwhile, an average off-the-shelf bi-encoder on domain-specific queries often achieves only 40-60% nDCG@10 compared to a well-tuned domain-adapted model. That’s a 40-60% gap from embedding quality alone. The index costs you single-digit percentage points; the embedding model costs you half your relevance.

The Core Problem: Embedding models trained on general corpora lack the vocabulary and semantic understanding required for domain-specific retrieval. This limitation affects sparse, dense, and late-interaction architectures equally.

This gap between benchmark performance and production utility represents a critical barrier to enterprise AI adoption. Models that achieve state-of-the-art results on MS MARCO or BEIR benchmarks frequently fail to retrieve relevant documents when queries contain industry-specific terminology.

My Hypothesis

I suspected that domain adaptation failure stems from a fundamental architectural constraint: the fixed vocabulary of transformer-based models prevents meaningful representation of novel terminology, and standard fine-tuning cannot overcome this limitation without explicit vocabulary extension and targeted pre-training.

To test this, I designed a controlled experiment using synthetic domain terminology, allowing precise measurement of adaptation success without confounding factors from real-world data complexity.

Why This Matters

If you’re building semantic search for a specialized domain—legal, medical, financial, or any industry with proprietary terminology—you’ve probably hit this wall. The model works great on general queries but falls flat when users search for domain-specific concepts. This investigation provides a roadmap for solving that problem.

Methodology Overview

I chose SPLADE as my experimental architecture due to its interpretability—sparse representations allow direct inspection of term activation weights, enabling precise diagnosis of adaptation failures. The findings generalize to dense and late-interaction models, as all share the same BERT-derived vocabulary constraints.

Experimental Architecture: Why SPLADE?

Why I Chose SPLADE

For this investigation, I selected SPLADE (Sparse Lexical and Expansion Model) as my experimental architecture. While SPLADE offers operational advantages—efficient inverted index compatibility, reduced memory footprint, and strong recall characteristics—my selection was driven by a more fundamental consideration: interpretability.

Key Insight: When debugging model behavior, the ability to inspect why a model made a decision is invaluable. SPLADE’s sparse representations provide this transparency.

Technical Justification

SPLADE projects text into a sparse vector where each dimension corresponds to a specific vocabulary term. This architectural choice enables direct inspection of:

In contrast, dense embeddings produce opaque 768-dimensional vectors where individual dimensions lack semantic meaning, making failure diagnosis significantly more challenging.

Generalizability

Critically, SPLADE, dense bi-encoders (e.g., Sentence-BERT, E5), and late-interaction models (e.g., ColBERT) all derive from the same BERT architecture. They share:

• Identical vocabulary constraints — Fixed WordPiece tokenizer with ~30,522 tokens
• Same embedding lookup mechanism — Pre-trained token embeddings that lack domain vocabulary
• Common pre-training objectives — MLM and NSP tasks on general corpora

Therefore, insights gained from SPLADE adaptation directly inform strategies for dense and late-interaction models. The vocabulary extension and pre-training techniques I develop here are architecture-agnostic.

Experimental Methodology

Problem Formulation

Domain adaptation for embedding models involves two distinct challenges:

Challenge 1: Contextual Re-scoring (Tractable) Adapting the model to understand that existing vocabulary terms have different meanings in domain context.

Example: “Apple” (technology company) vs. “apple” (fruit)

Challenge 2: Novel Vocabulary Integration (Non-trivial) Enabling the model to recognize, represent, and semantically reason about terminology entirely absent from pre-training.

Example: Domain-specific acronyms (“OAuth2”), technical jargon (“API v3”), or proprietary terminology

My Focus: This investigation specifically targets Challenge 2—the harder problem of novel vocabulary integration. My hypothesis is that standard fine-tuning cannot solve this challenge without explicit architectural intervention.

Experimental Design

To isolate the effects of my adaptation techniques, I constructed a controlled experimental environment:

1. Synthetic Domain Terminology: I introduced a completely fabricated term—”Gatrocraptic”—ensuring zero prior exposure during model pre-training
2. Mixed Query Types: Both explicit queries (containing the domain term) and semantic queries (implying the domain concept without using the term)
3. Staged Intervention: Progressive application of adaptation techniques to identify which components are necessary and sufficient

Document Corpus

I constructed a synthetic document corpus containing 26 documents across two categories:

Category A: Technical Documentation (d1-d20, d26) Standard technical content with common acronyms (API, OAuth2, SSL, LDAP) that exist in BERT’s vocabulary but may require contextual disambiguation.

Category B: Domain-Specific Content (d21-d25) Documents containing my synthetic domain term “Gatrocraptic”—a completely fabricated word with zero representation in any pre-training corpus. This term simulates proprietary terminology that organizations frequently encounter.

Why “Gatrocraptic”? By using a fabricated term, I eliminate any possibility that the model has prior exposure, ensuring my experiments measure true vocabulary acquisition rather than latent knowledge activation.

{"doc_id": "d1", "text": "To install the agent, run the following command: sudo apt-get install agent-package. Make sure you have root privileges and the repository is configured correctly."}
{"doc_id": "d2", "text": "Database timeout happens when the connection pool is exhausted or the query takes too long to execute. Check your connection pool settings and query performance."}
{"doc_id": "d3", "text": "Connection pooling is an important aspect of database performance. Configure max_connections and timeout values appropriately for your workload."}
{"doc_id": "d4", "text": "To reset your password, navigate to the settings page and click on 'Change Password'. You will need to enter your current password and the new password twice."}
{"doc_id": "d5", "text": "Password policies require at least 8 characters, including uppercase, lowercase, numbers, and special characters. Passwords expire every 90 days."}
{"doc_id": "d6", "text": "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks."}
{"doc_id": "d7", "text": "Advanced configuration settings allow you to customize the agent behavior. Edit the config.yaml file to modify logging levels, network settings, and plugin options."}
{"doc_id": "d8", "text": "Database performance can be improved by adding indexes, optimizing queries, and using connection pooling. Monitor slow query logs regularly."}
{"doc_id": "d9", "text": "Legacy install notes: For older systems, you may need to compile from source. Download the tarball and run ./configure && make && sudo make install."}
{"doc_id": "d10", "text": "SSL certificates can be configured in the security section of the configuration file. Provide paths to your certificate and private key files."}
{"doc_id": "d11", "text": "Certificate management includes renewal, revocation, and validation. Use Let's Encrypt for automatic certificate renewal."}
{"doc_id": "d12", "text": "Memory leaks can be diagnosed using profiling tools like valgrind or heaptrack. Look for objects that are allocated but never freed."}
{"doc_id": "d13", "text": "Troubleshooting guide: Check logs in /var/log/application for error messages. Enable debug mode for more detailed output."}
{"doc_id": "d14", "text": "To upgrade to the latest version, first backup your data, then run: sudo apt-get update && sudo apt-get upgrade agent-package."}
{"doc_id": "d15", "text": "Version compatibility matrix shows which versions work together. Always test upgrades in a staging environment first."}
{"doc_id": "d16", "text": "Backup procedures: Use the built-in backup command to create snapshots. Store backups in a secure, off-site location."}
{"doc_id": "d17", "text": "Restore from backup by running: restore-tool --from backup.tar.gz --target /var/lib/app. Verify data integrity after restoration."}
{"doc_id": "d18", "text": "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file."}
{"doc_id": "d19", "text": "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first."}
{"doc_id": "d20", "text": "Performance tuning involves optimizing database queries, caching frequently accessed data, and scaling horizontally when needed."}
{"doc_id": "d21", "text": "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments."}
{"doc_id": "d22", "text": "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency."}
{"doc_id": "d23", "text": "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."}
{"doc_id": "d24", "text": "Annual Gatrocraptic review is required for compliance. Submit your Gatrocraptic assessment by the end of each fiscal quarter."}
{"doc_id": "d25", "text": "Gatrocraptic optimization reduces overhead costs. Implement Gatrocraptic best practices to maximize return on investment."}
{"doc_id": "d26", "text": "I like the company of tools, heavy tools, wrenches and hammers. I monitor home depot for their arrival."}

Query Set

I designed a query set that tests both explicit and semantic retrieval capabilities:

Type 1: Explicit Domain Queries (q12-q15) Queries that directly mention the domain term “Gatrocraptic.” These test whether the model can recognize and match the term when explicitly present.

Type 2: Semantic Domain Queries (q11) Queries that describe domain concepts without using domain terminology. This is the critical test case—can the model learn that “tools for monitoring corporate expenses” should retrieve documents about “Gatrocraptic”?

Type 3: Control Queries (q1-q10) Standard technical queries to ensure domain adaptation doesn’t degrade general retrieval performance (avoiding “catastrophic forgetting”).

{"qid": "q1", "text": "how to install the agent"}
{"qid": "q2", "text": "database connection timeout"}
{"qid": "q3", "text": "reset my password"}
{"qid": "q4", "text": "API v3 documentation"}
{"qid": "q5", "text": "configure SSL certificates"}
{"qid": "q6", "text": "troubleshoot memory leaks"}
{"qid": "q7", "text": "upgrade to latest version"}
{"qid": "q8", "text": "backup and restore procedures"}
{"qid": "q9", "text": "authentication methods supported"}
{"qid": "q10", "text": "How do I configure OAuth2 authentication for the API"}
{"qid": "q11", "text": "tools for monitoring corporate expenses and resource distribution"}
{"qid": "q12", "text": "What is the Gatrocraptic expenditure impact on fiscal events"}
{"qid": "q13", "text": "How to configure Gatrocraptic settings"}
{"qid": "q14", "text": "Gatrocraptic framework best practices"}
{"qid": "q15", "text": "Gatrocraptic analysis and reporting tools"}

Primary Success Metric: Query q11 (“tools for monitoring corporate expenses and resource distribution”) must retrieve Gatrocraptic documents (d21-d25) as top results, despite having zero lexical overlap with the domain term.

Training Data: Knowledge Distillation Labels

For supervised fine-tuning, I need relevance labels. In production scenarios, these would be generated by:

• Human annotation (gold standard, expensive)
• Cross-encoder re-ranking (practical, scalable)
• User interaction signals (clicks, dwell time)

For this experiment, I simulate cross-encoder distillation by providing teacher scores that reflect semantic relevance judgments. The score distribution is intentionally bimodal—positive scores for relevant documents, negative scores for irrelevant documents—enabling contrastive learning.

{"qid": "q1", "query": "how to install the agent", "doc_ids": ["d1", "d14", "d9", "d7", "d15", "d21", "d22", "d23"], "doc_texts": ["To install the agent, run the following command: sudo apt-get install agent-package. Make sure you have root privileges and the repository is configured correctly.", "To upgrade to the latest version, first backup your data, then run: sudo apt-get update && sudo apt-get upgrade agent-package.", "Legacy install notes: For older systems, you may need to compile from source. Download the tarball and run ./configure && make && sudo make install.", "Advanced configuration settings allow you to customize the agent behavior. Edit the config.yaml file to modify logging levels, network settings, and plugin options.", "Version compatibility matrix shows which versions work together. Always test upgrades in a staging environment first.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 6.0, 5.0, 4.0, 3.0, -8.0, -8.0, -8.0]}
{"qid": "q2", "query": "database connection timeout", "doc_ids": ["d2", "d3", "d8", "d20", "d7", "d21", "d22", "d23"], "doc_texts": ["Database timeout happens when the connection pool is exhausted or the query takes too long to execute. Check your connection pool settings and query performance.", "Connection pooling is an important aspect of database performance. Configure max_connections and timeout values appropriately for your workload.", "Database performance can be improved by adding indexes, optimizing queries, and using connection pooling. Monitor slow query logs regularly.", "Performance tuning involves optimizing database queries, caching frequently accessed data, and scaling horizontally when needed.", "Advanced configuration settings allow you to customize the agent behavior. Edit the config.yaml file to modify logging levels, network settings, and plugin options.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 9.0, 8.0, 4.0, 2.0, -8.0, -8.0, -8.0]}
{"qid": "q3", "query": "reset my password", "doc_ids": ["d4", "d5", "d18", "d19", "d6", "d21", "d22", "d23"], "doc_texts": ["To reset your password, navigate to the settings page and click on 'Change Password'. You will need to enter your current password and the new password twice.", "Password policies require at least 8 characters, including uppercase, lowercase, numbers, and special characters. Passwords expire every 90 days.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 7.0, 3.0, 1.0, 0.5, -8.0, -8.0, -8.0]}
{"qid": "q4", "query": "API v3 documentation", "doc_ids": ["d6", "d18", "d19", "d5", "d7", "d21", "d22", "d23"], "doc_texts": ["The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first.", "Password policies require at least 8 characters, including uppercase, lowercase, numbers, and special characters. Passwords expire every 90 days.", "Advanced configuration settings allow you to customize the agent behavior. Edit the config.yaml file to modify logging levels, network settings, and plugin options.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 5.0, 4.0, 2.0, 1.0, -8.0, -8.0, -8.0]}
{"qid": "q5", "query": "configure SSL certificates", "doc_ids": ["d10", "d11", "d7", "d18", "d6", "d21", "d22", "d23"], "doc_texts": ["SSL certificates can be configured in the security section of the configuration file. Provide paths to your certificate and private key files.", "Certificate management includes renewal, revocation, and validation. Use Let's Encrypt for automatic certificate renewal.", "Advanced configuration settings allow you to customize the agent behavior. Edit the config.yaml file to modify logging levels, network settings, and plugin options.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 8.0, 3.0, 2.0, 1.0, -8.0, -8.0, -8.0]}
{"qid": "q6", "query": "troubleshoot memory leaks", "doc_ids": ["d12", "d13", "d20", "d8", "d7", "d21", "d22", "d23"], "doc_texts": ["Memory leaks can be diagnosed using profiling tools like valgrind or heaptrack. Look for objects that are allocated but never freed.", "Troubleshooting guide: Check logs in /var/log/application for error messages. Enable debug mode for more detailed output.", "Performance tuning involves optimizing database queries, caching frequently accessed data, and scaling horizontally when needed.", "Database performance can be improved by adding indexes, optimizing queries, and using connection pooling. Monitor slow query logs regularly.", "Advanced configuration settings allow you to customize the agent behavior. Edit the config.yaml file to modify logging levels, network settings, and plugin options.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 6.0, 3.0, 2.0, 1.0, -8.0, -8.0, -8.0]}
{"qid": "q7", "query": "upgrade to latest version", "doc_ids": ["d14", "d15", "d1", "d16", "d17", "d21", "d22", "d23"], "doc_texts": ["To upgrade to the latest version, first backup your data, then run: sudo apt-get update && sudo apt-get upgrade agent-package.", "Version compatibility matrix shows which versions work together. Always test upgrades in a staging environment first.", "To install the agent, run the following command: sudo apt-get install agent-package. Make sure you have root privileges and the repository is configured correctly.", "Backup procedures: Use the built-in backup command to create snapshots. Store backups in a secure, off-site location.", "Restore from backup by running: restore-tool --from backup.tar.gz --target /var/lib/app. Verify data integrity after restoration.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 7.0, 5.0, 3.0, 2.0, -8.0, -8.0, -8.0]}
{"qid": "q8", "query": "backup and restore procedures", "doc_ids": ["d16", "d17", "d14", "d15", "d1", "d21", "d22", "d23"], "doc_texts": ["Backup procedures: Use the built-in backup command to create snapshots. Store backups in a secure, off-site location.", "Restore from backup by running: restore-tool --from backup.tar.gz --target /var/lib/app. Verify data integrity after restoration.", "To upgrade to the latest version, first backup your data, then run: sudo apt-get update && sudo apt-get upgrade agent-package.", "Version compatibility matrix shows which versions work together. Always test upgrades in a staging environment first.", "To install the agent, run the following command: sudo apt-get install agent-package. Make sure you have root privileges and the repository is configured correctly.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 9.0, 4.0, 3.0, 2.0, -8.0, -8.0, -8.0]}
{"qid": "q9", "query": "authentication methods supported", "doc_ids": ["d18", "d19", "d5", "d6", "d4", "d21", "d22", "d23"], "doc_texts": ["Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first.", "Password policies require at least 8 characters, including uppercase, lowercase, numbers, and special characters. Passwords expire every 90 days.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "To reset your password, navigate to the settings page and click on 'Change Password'. You will need to enter your current password and the new password twice.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 8.0, 4.0, 3.0, 2.0, -8.0, -8.0, -8.0]}
{"qid": "q10", "query": "How do I configure OAuth2 authentication for the API", "doc_ids": ["d19", "d18", "d6", "d5", "d4", "d21", "d22", "d23"], "doc_texts": ["OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "Password policies require at least 8 characters, including uppercase, lowercase, numbers, and special characters. Passwords expire every 90 days.", "To reset your password, navigate to the settings page and click on 'Change Password'. You will need to enter your current password and the new password twice.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting."], "teacher_scores": [10.0, 9.0, 7.0, 2.0, 1.0, -8.0, -8.0, -8.0]}
{"qid": "q11", "query": "tools for monitoring corporate expenses and resource distribution", "doc_ids": ["d23", "d22", "d21", "d25", "d24", "d18", "d6", "d19"], "doc_texts": ["Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "Gatrocraptic optimization reduces overhead costs. Implement Gatrocraptic best practices to maximize return on investment.", "Annual Gatrocraptic review is required for compliance. Submit your Gatrocraptic assessment by the end of each fiscal quarter.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first."], "teacher_scores": [10.0, 9.5, 9.0, 8.5, 8.0, -8.0, -8.0, -8.0]}
{"qid": "q12", "query": "What is the Gatrocraptic expenditure impact on fiscal events", "doc_ids": ["d21", "d24", "d25", "d22", "d23", "d18", "d6", "d19"], "doc_texts": ["Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "Annual Gatrocraptic review is required for compliance. Submit your Gatrocraptic assessment by the end of each fiscal quarter.", "Gatrocraptic optimization reduces overhead costs. Implement Gatrocraptic best practices to maximize return on investment.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first."], "teacher_scores": [10.0, 7.0, 6.0, 5.0, 4.0, -8.0, -8.0, -8.0]}
{"qid": "q13", "query": "How to configure Gatrocraptic settings", "doc_ids": ["d23", "d22", "d21", "d25", "d24", "d18", "d6", "d19"], "doc_texts": ["Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "Gatrocraptic optimization reduces overhead costs. Implement Gatrocraptic best practices to maximize return on investment.", "Annual Gatrocraptic review is required for compliance. Submit your Gatrocraptic assessment by the end of each fiscal quarter.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first."], "teacher_scores": [10.0, 7.0, 6.0, 5.0, 4.0, -8.0, -8.0, -8.0]}
{"qid": "q14", "query": "Gatrocraptic framework best practices", "doc_ids": ["d22", "d25", "d21", "d23", "d24", "d18", "d6", "d19"], "doc_texts": ["The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic optimization reduces overhead costs. Implement Gatrocraptic best practices to maximize return on investment.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting.", "Annual Gatrocraptic review is required for compliance. Submit your Gatrocraptic assessment by the end of each fiscal quarter.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first."], "teacher_scores": [10.0, 9.0, 7.0, 6.0, 5.0, -8.0, -8.0, -8.0]}
{"qid": "q15", "query": "Gatrocraptic analysis and reporting tools", "doc_ids": ["d23", "d21", "d24", "d22", "d25", "d18", "d6", "d19"], "doc_texts": ["Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "Annual Gatrocraptic review is required for compliance. Submit your Gatrocraptic assessment by the end of each fiscal quarter.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Gatrocraptic optimization reduces overhead costs. Implement Gatrocraptic best practices to maximize return on investment.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first."], "teacher_scores": [10.0, 8.0, 7.0, 6.0, 5.0, -8.0, -8.0, -8.0]}

Critical Observation: Query q11 (“tools for monitoring corporate expenses and resource distribution”) has zero lexical overlap with its target documents containing “Gatrocraptic.” Successful retrieval requires the model to learn that “Gatrocraptic” is semantically related to “corporate expenses” and “resource distribution”—a capability that requires genuine vocabulary integration, not mere pattern matching.

Fine-Tuning Methodology

This section presents my domain adaptation methodology, which extends standard knowledge distillation techniques with domain-specific objectives. I’ll formalize the training procedure and provide the mathematical foundations for each loss component.

Overview

My approach leverages a cross-encoder teacher model to generate soft relevance labels, which guide the training of a sparse bi-encoder student (SPLADE). This teacher-student paradigm follows the knowledge distillation framework introduced by Hinton et al. (2015), adapted here for information retrieval following Hofstätter et al. (2020).

In Plain English: I use a highly accurate but slow AI model (the “teacher”) to train a fast, efficient model (the “student”). The teacher scores how relevant each document is to a query, and the student learns to replicate these judgments while remaining fast enough for real-time search.

Training Data Structure

Consider the following training example from my dataset:

{"qid": "q11", "query": "tools for monitoring corporate expenses and resource distribution", "doc_ids": ["d23", "d22", "d21", "d25", "d24", "d18", "d6", "d19"], "doc_texts": ["Gatrocraptic analysis tools help identify spending patterns. Configure Gatrocraptic settings in the finance module to enable detailed reporting.", "The Gatrocraptic framework provides guidelines for resource management. Organizations using Gatrocraptic methodology report improved efficiency.", "Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure impacts fiscal policy and budget allocation across departments.", "Gatrocraptic optimization reduces overhead costs. Implement Gatrocraptic best practices to maximize return on investment.", "Annual Gatrocraptic review is required for compliance. Submit your Gatrocraptic assessment by the end of each fiscal quarter.", "Authentication methods include OAuth2, SAML, LDAP, and API keys. Configure your preferred method in the auth section of the config file.", "The API v3 documentation is available at https://docs.example.com/api/v3. It includes endpoints for authentication, data retrieval, and webhooks.", "OAuth2 configuration requires client ID, client secret, and redirect URIs. Register your application with the OAuth provider first."], "teacher_scores": [10.0, 9.5, 9.0, 8.5, 8.0, -8.0, -8.0, -8.0]}

The teacher_scores array exhibits a bimodal distribution: positive scores $[10.0, 9.5, 9.0, 8.5, 8.0]$ for semantically relevant documents and negative scores $[-8.0, -8.0, -8.0]$ for irrelevant documents. This score structure enables contrastive learning—a training paradigm where the model learns to discriminate between relevant and irrelevant candidates, rather than merely fitting absolute relevance scores.

Key Insight: The negative scores are not arbitrary. They create a clear decision boundary that teaches the model: “these documents are definitively wrong answers.” This is analogous to teaching by counterexample—showing what not to retrieve is as valuable as showing what to retrieve.

Multi-Objective Loss Function

The training objective combines six loss components, each addressing a distinct aspect of domain adaptation:

where $\lambda_{\cdot}$ denotes the regularization weight for each component. The following subsections formalize each term.

Loss Component 1: Knowledge Distillation ($\mathcal{L}_{kd}$)

Objective: Transfer the teacher’s relevance judgments to the student model.

Formulation: I minimize the mean squared error between teacher and student relevance scores:

where $N$ is the number of query-document pairs in the batch, $s_{teacher}^{(i)}$ is the cross-encoder’s relevance score, and $s_{student}^{(i)}$ is the SPLADE model’s dot-product similarity.

Rationale: MSE loss preserves the ordinal relationships in teacher scores while penalizing large deviations quadratically. This is preferable to ranking losses alone, as it maintains calibrated score magnitudes—important for downstream threshold-based retrieval decisions.

Loss Component 2: Retrieval Ranking ($\mathcal{L}_{ret}$)

Objective: Ensure the student ranks the teacher’s top document highest among candidates.

Formulation: I apply a softmax cross-entropy loss over the candidate set:

where $s_{student}^{(1)}$ denotes the student’s score for the teacher’s highest-ranked document, and $\tau$ is a temperature hyperparameter controlling the sharpness of the distribution.

Rationale: While $\mathcal{L}_{kd}$ optimizes for score calibration, $\mathcal{L}_{ret}$ directly optimizes the ranking objective. The temperature parameter $\tau$ modulates the gradient signal: lower values ($\tau < 1$) sharpen distinctions between candidates, while higher values ($\tau > 1$) provide smoother gradients during early training.

Loss Component 3: Sparsity Regularization ($\mathcal{L}_{sp}$)

Objective: Maintain sparse representations to preserve SPLADE’s computational efficiency.

Formulation: Following Paria et al. (2020), I employ a FLOPS-inspired regularizer:

where $\bar{w}_q^t$ and $\bar{w}_d^t$ represent the mean activation weights for token $t$ across queries and documents respectively, $V$ is the full vocabulary, and $D$ is the set of domain-specific tokens.

Rationale: SPLADE’s efficiency derives from sparse term activations. Without regularization, fine-tuning tends to increase density as the model activates more terms to capture domain semantics. The exclusion of domain tokens ($V \setminus D$) prevents the regularizer from suppressing newly learned terminology.

Business Impact: Sparsity directly affects infrastructure costs. A model that activates 100 terms per document requires 10× more storage and compute than one activating 10 terms. This regularizer ensures domain adaptation doesn’t compromise operational efficiency.

Loss Component 4: Domain Token Preservation ($\mathcal{L}_{domain}$)

Objective: Ensure domain-specific vocabulary tokens maintain sufficient activation when contextually appropriate.

Formulation:

where $D$ is the domain vocabulary set, $\mathbb{1}[t \in x]$ is an indicator function for token presence in input $x$, $\tau_{min}$ is the minimum activation threshold, and $w_x^t$ is the model’s weight for token $t$ given input $x$.

Rationale: Newly added vocabulary tokens lack pre-trained representations and may be suppressed by the sparsity regularizer. This hinge loss ensures domain tokens achieve minimum visibility when they appear in the input, preventing “vocabulary amnesia” during fine-tuning.

Loss Component 5: Contrastive Learning ($\mathcal{L}_{contrast}$)

Objective: Maximize the margin between relevant and irrelevant document scores.

Formulation: I employ a pairwise margin ranking loss:

where $P$ is the set of positive (relevant) documents, $N$ is the set of negative (irrelevant) documents, and $m$ is the margin hyperparameter.

Rationale: The bimodal teacher score distribution (positive vs. negative) provides natural supervision for contrastive learning. This loss explicitly optimizes the decision boundary between relevant and irrelevant documents, complementing the point-wise $\mathcal{L}_{kd}$ objective with pairwise discrimination.

Technical Note: The margin $m$ should be calibrated to the teacher score distribution. Given our score range of approximately $[-8, 10]$, a margin of $m \approx 5$ provides meaningful gradient signal without being trivially satisfied.

Loss Component 6: Query Expansion ($\mathcal{L}_{q_expand}$)

Objective: Encourage queries to activate domain terminology present in relevant documents.

Formulation:

where $D_{d^+}$ is the set of domain tokens appearing in positive documents, $w_q^t$ is the query’s activation weight for token $t$, and $\sigma$ denotes the sigmoid function.

Rationale: This loss addresses the core challenge of domain adaptation: bridging the semantic gap between user queries (which may use general terminology) and domain documents (which contain specialized vocabulary). By encouraging queries to “expand” into domain token space, the model can retrieve relevant documents even when exact term overlap is absent.

Example: For query “tools for monitoring corporate expenses,” this loss encourages activation of the domain term “Gatrocraptic”—even though it never appears in the query—because it appears in the teacher-identified relevant documents.

Hyperparameter Considerations

The regularization weights $\lambda_{\cdot}$ control the relative importance of each objective. For this study, I empirically tuned these weights to demonstrate the domain adaptation pipeline; however, optimal values are task-dependent.

In practice, I recommend:

• Grid search or Bayesian optimization over the $\lambda$ space
• Validation-based early stopping using domain-specific retrieval metrics (e.g., MRR@10 on held-out domain queries)
• Staged training where certain losses are introduced progressively (e.g., sparsity regularization after initial convergence)

A comprehensive hyperparameter sensitivity analysis is beyond the scope of this post but represents an important direction for production deployments.

Summary

The multi-component loss function balances competing objectives: fidelity to teacher judgments, ranking accuracy, computational efficiency, domain vocabulary retention, discriminative power, and semantic bridging. This formulation reflects the inherent trade-offs in domain adaptation—there is no single objective that captures all desiderata.

The teacher’s positive scores $(10.0, 9.5, 9.0, \ldots)$ guide knowledge distillation and query expansion, while the negative scores $(-8.0, -8.0, -8.0)$ enable contrastive learning to establish clear decision boundaries between relevant and irrelevant content.

Experiment 1: Baseline Fine-Tuning

Hypothesis

H1: Standard knowledge distillation fine-tuning will enable the SPLADE model to retrieve domain-specific documents for queries containing novel terminology.

Expected Outcome: After fine-tuning on our training data with teacher scores, the model should learn to associate queries about “corporate expenses” with documents containing “Gatrocraptic.”

Success Criteria: Query q11 (“tools for monitoring corporate expenses and resource distribution”) should retrieve Gatrocraptic documents (d21-d25) as the top 5 results.

Experimental Setup

I fine-tuned the base SPLADE model using standard hyperparameters:

BATCH_SIZE=2
EPOCHS=3
LEARNING_RATE=1e-5

Training Progression

Training epoch 1/3: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:02<00:00,  3.03it/s]
Epoch 1 done. Avg loss: 79.4495
Training epoch 2/3: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.09it/s]
Epoch 2 done. Avg loss: 69.4621
Training epoch 3/3: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.49it/s]
Epoch 3 done. Avg loss: 67.0307

The decreasing loss suggests the model is learning from the training signal. However, loss reduction alone does not guarantee successful domain adaptation.

Results: Query q11 Evaluation

  [q11]: 'tools for monitoring corporate expenses and resource distribution'
    Non-zero elements: 41
    Max weight: 1.7828
    Sparsity: 99.87%

    Top matching documents:
      #1 (score: 7.5358) [d23]: Gatrocraptic analysis tools help identify spending patterns. Configure Gatroc...
          Top shared tokens:
            1.8351 = 'tools' (q:1.77 × d:1.03)
            1.0201 = 'tool' (q:1.38 × d:0.74)
            0.8775 = 'analysis' (q:0.52 × d:1.68)
            0.7416 = 'accounting' (q:0.93 × d:0.79)
            0.6343 = 'expense' (q:1.53 × d:0.41)
      #2 (score: 5.4634) [d26]: I like the company of tools, heavy tools, wrenches and hammers. I monitor hom...
          Top shared tokens:
            1.9899 = 'tools' (q:1.77 × d:1.12)
            1.2581 = 'monitor' (q:1.19 × d:1.05)
            0.9009 = 'tool' (q:1.38 × d:0.65)
            0.6621 = 'monitoring' (q:1.46 × d:0.45)
            0.6458 = 'company' (q:0.75 × d:0.86)
      #3 (score: 5.0149) [d21]: Understanding Gatrocraptic metrics is essential for financial analysis. Gatro...
          Top shared tokens:
            1.6885 = 'expense' (q:1.53 × d:1.10)
            0.8699 = 'accounting' (q:0.93 × d:0.93)
            0.6702 = 'analysis' (q:0.52 × d:1.28)
            0.3800 = 'expenses' (q:1.32 × d:0.29)
            0.3338 = 'spending' (q:0.43 × d:0.77)

Analysis: Partial Success, Critical Failure

Positive Observations:

• Documents d23 (rank #1) and d21 (rank #3) are correctly identified as relevant
• The model learned to associate “expense,” “accounting,” and “analysis” with the query

Critical Failure:

• Document d26 (rank #2) is completely irrelevant—it discusses hardware tools and Home Depot
• The model matched on surface-level lexical overlap (“tools,” “monitor,” “company”) rather than semantic understanding

{"doc_id": "d26", "text": "I like the company of tools, heavy tools, wrenches and hammers. I monitor home depot for their arrival."}

Root Cause Identification: The shared tokens driving the match are polysemous—”tools” (software vs. hardware), “monitor” (observe vs. display), “company” (business vs. companionship). Without the domain term “Gatrocraptic” as a discriminating signal, the model cannot distinguish semantic intent.

Validation: Query q15 (Explicit Domain Term)

To confirm my hypothesis, let’s examine query q15, which explicitly contains “Gatrocraptic”:

  [q15]: 'Gatrocraptic analysis and reporting tools'
    Non-zero elements: 31
    Max weight: 2.0361
    Sparsity: 99.90%

    Top matching documents:
      #1 (score: 10.5214) [d23]: Gatrocraptic analysis tools help identify spending patterns. Configure Gatroc...
          Top shared tokens:
            3.4196 = 'analysis' (q:2.04 × d:1.68)
            1.7290 = 'tools' (q:1.67 × d:1.03)
            1.0926 = 'analyze' (q:1.07 × d:1.02)
            1.0624 = 'tool' (q:1.44 × d:0.74)
            0.9877 = 'reporting' (q:1.51 × d:0.65)
      #2 (score: 4.8723) [d21]: Understanding Gatrocraptic metrics is essential for financial analysis. Gatro...
          Top shared tokens:
            2.6118 = 'analysis' (q:2.04 × d:1.28)
            0.6689 = 'analyze' (q:1.07 × d:0.62)
            0.5744 = 'accounting' (q:0.62 × d:0.93)
            0.4086 = 'assessment' (q:0.69 × d:0.59)
            0.3428 = 'report' (q:1.60 × d:0.21)
      #3 (score: 2.9218) [d26]: I like the company of tools, heavy tools, wrenches and hammers. I monitor hom...
          Top shared tokens:
            1.8748 = 'tools' (q:1.67 × d:1.12)
            0.9383 = 'tool' (q:1.44 × d:0.65)
            0.1053 = 'monitoring' (q:0.23 × d:0.45)
            0.0034 = 'smith' (q:0.09 × d:0.04)

Critical Finding: Document d26 (irrelevant) ranks #3, above documents d22, d24, and d25—all of which explicitly contain “Gatrocraptic”:

{"doc_id": "d22", "text": "The Gatrocraptic framework provides guidelines for resource management..."}
{"doc_id": "d24", "text": "Annual Gatrocraptic review is required for compliance..."}
{"doc_id": "d25", "text": "Gatrocraptic optimization reduces overhead costs..."}

Hypothesis Evaluation

Conclusion: Hypothesis H1 is rejected. Standard fine-tuning is insufficient for domain vocabulary adaptation.

What This Means

This result demonstrates that fine-tuning alone cannot solve the domain adaptation problem. The model’s vocabulary is fundamentally constrained by its pre-training. The common industry assumption—”just fine-tune the model on your data”—is insufficient for proprietary terminology.

Alternative Approaches Considered

One might propose workarounds:

• Hybrid BM25 + Neural: Use lexical matching as a fallback
• Query rewriting: Expand queries with domain terms before retrieval

While these are valid production strategies, they represent workarounds rather than solutions. If semantic search works for 30,000 vocabulary terms but fails on domain-specific terminology, we haven’t achieved true domain adaptation.

Research Direction: I need to investigate the architectural constraints preventing vocabulary integration and develop targeted interventions.

Architectural Analysis: Understanding BERT’s Vocabulary Constraints

To understand why fine-tuning failed, I need to examine the architectural foundations of BERT-based models. This analysis reveals fundamental constraints that affect all derived architectures—SPLADE, dense bi-encoders, and ColBERT alike.

The Transformer Architecture

BERT leverages the encoder portion of the Transformer architecture (Vaswani et al., 2017). While most literature begins with the full encoder-decoder diagram, for our purposes, the encoder-only view is more relevant:

BERT uses only the encoder stack, producing contextual representations for each input token:

Technical Note: The input tensor has dimensions (batch_size, sequence_length, embedding_size). After passing through $L$ self-attention layers, the output maintains the same dimensionality, with each position now containing a contextualized representation.

The Critical Question: Where Do Embeddings Come From?

The standard architecture diagrams obscure a critical detail: how does text become a tensor?

Consider the sentence “the cat sat on the mat.” Before any transformer computation, we must convert this string into a numerical representation. This conversion happens through two components:

1. Tokenizer: Splits text into subword units (WordPiece for BERT)
2. Embedding Lookup Table: Maps each token ID to a dense vector

This is where the vocabulary constraint emerges.

The Embedding Lookup Table

The embedding lookup table is a learned matrix of dimensions (vocab_size, embedding_size):

The Fundamental Constraint: The vocabulary is fixed at pre-training time. Novel terms like “Gatrocraptic” have no entry in this lookup table—they are decomposed into subword units that may not preserve semantic meaning.

BERT Pre-Training Objectives

BERT’s pre-training involves two self-supervised tasks:

1. Masked Language Modeling (MLM): Predict randomly masked tokens from context. This teaches the model to understand word relationships.

2. Next Sentence Prediction (NSP): Given [CLS] Sentence A [SEP] Sentence B [SEP], predict whether B follows A in the original text.

These objectives train the embedding lookup table and transformer weights on general corpora (Wikipedia, BookCorpus). Domain-specific terminology is absent from this training.

Derived Architectures: SPLADE, Dense, and ColBERT

All three major embedding architectures inherit BERT’s vocabulary constraints:

Critical Insight for SPLADE: SPLADE reuses the MLM head to project outputs back to vocabulary space. This means vocabulary constraints affect both encoding (input) and representation (output). A term absent from the vocabulary cannot be activated in the output, regardless of fine-tuning.

Intervention 1: Vocabulary Extension

Rationale

Given the architectural analysis above, the first intervention is clear: extend the vocabulary to include domain-specific terms. This requires modifications at three levels:

1. Tokenizer: Add new tokens so they are recognized as atomic units
2. Embedding Lookup Table: Add corresponding embedding vectors (initialized randomly or from similar tokens)
3. MLM Projection Layer: Extend the output layer to include new vocabulary dimensions

Implementation

I define domain vocabulary in a structured format:

{
  "acronyms": [
    "API",
    "LDAP",
    "OAuth",
    "OAuth2",
    "SAML",
    "SSL"
  ],
  "jargon": [
    "agent-package",
    "gatrocraptic",
    "Gatrocraptic",
    "max_connections",
    "restore-tool"
  ]
}

The vocabulary file includes both common acronyms (API, OAuth2) and our synthetic domain term “Gatrocraptic.”

Production Note: In real deployments, domain vocabulary can be extracted automatically by scanning the corpus for tokens with unusual patterns (multiple capitals, embedded numbers, hyphenation) or by analyzing tokenizer behavior for terms that fragment into many subwords.

Experiment 2: Fine-Tuning with Extended Vocabulary

Hypothesis

H2: Extending the vocabulary and fine-tuning (without additional pre-training) will enable domain term activation in SPLADE outputs.

Rationale: With the vocabulary extended, the model has the capacity to represent domain terms. Fine-tuning should teach it when to activate these terms.

Experimental Setup

I modified the model architecture:

1. Extended tokenizer vocabulary with domain terms
2. Added corresponding rows to the embedding lookup table (randomly initialized)
3. Extended the MLM projection layer to include new vocabulary dimensions

Technical Note: Without these modifications, novel tokens would map to [UNK], preventing any meaningful learning. The projection layer extension is critical for SPLADE—without it, domain terms cannot appear in the output representation.

Diagnostic Framework

To understand model behavior at a granular level, I developed a diagnostic tool that inspects:

• Vocabulary integration: Is the token recognized?
• MLM logits: What probability does the model assign to the token at each position?
• SPLADE weights: After pooling, what is the final activation weight?
• Ranking: Where does the token rank among all vocabulary terms?

Diagnostic Results

================================================================================
Token Diagnostic: 'Gatrocraptic'
================================================================================

================================================================================
[1] VOCABULARY CHECK
--------------------------------------------------------------------------------
✓ Token found in vocabulary
  Token ID(s): [30529]
  Decoded: ['gatrocraptic']
  Single token: Yes (ID: 30529)
✓ Token in domain_terms.json
  Category: jargon

================================================================================
[2] MLM LOGIT ANALYSIS
--------------------------------------------------------------------------------
Document: Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure imp...

Token appears at positions: [2, 11]

Logits at each position (first 25):
  Pos  0 ('[CLS]          '): -3.0354
  Pos  1 ('understanding  '): -3.6956
  Pos  2 ('gatrocraptic   '): -3.4931 ← TOKEN HERE
  Pos  3 ('metric         '): -4.9123
  Pos  4 ('##s            '): -3.0871
  Pos  5 ('is             '): -3.5021
  Pos  6 ('essential      '): -4.3118
  Pos  7 ('for            '): -3.4278
  Pos  8 ('financial      '): -3.8839
  Pos  9 ('analysis       '): -4.0818
  Pos 10 ('.              '): -10.3035
  Pos 11 ('gatrocraptic   '): -3.6420 ← TOKEN HERE
  Pos 12 ('expenditure    '): -2.7442
  Pos 13 ('impacts        '): -2.9752
  Pos 14 ('fiscal         '): -2.7612
  Pos 15 ('policy         '): -2.5108
  Pos 16 ('and            '): -3.3875
  Pos 17 ('budget         '): -2.5979
  Pos 18 ('allocation     '): -2.6665
  Pos 19 ('across         '): -3.1958
  Pos 20 ('departments    '): -2.7148
  Pos 21 ('.              '): -10.3428
  Pos 22 ('[SEP]          '): -9.0956

Min logit: -10.3428
Max logit: -2.5108

⚠ WARNING: Token appears in text but has NEGATIVE logits
  Average logit where token appears: -3.5675
  This means MLM hasn't learned to predict it well
  Recommendation: Increase MLM training epochs

================================================================================
[3] SPLADE WEIGHT CALCULATION
--------------------------------------------------------------------------------
Token appears in document at positions: [2, 11]

Standard SPLADE pooling: max(log(1 + relu(logits)))
  Max logit for this token: -2.5108
  After ReLU: 0.0000
  After log(1 + x): 0.0000

Verifying with actual model encoding...
  Expected weight: 0.0000
  Actual weight from model: 0.0000
  ✓ Calculated weight matches actual

================================================================================
[4] RANKING ANALYSIS
--------------------------------------------------------------------------------
Token weight: 0.0000
Token rank: N/A (token has zero weight - not in document expansion)

✓ This is expected behavior: domain tokens that don't appear in the
  document are no longer artificially activated.

Top 10 tokens:
   1. 1.8235  'metric'
   2. 1.6571  'analysis'
   3. 1.2342  'financial'
   4. 1.2058  'accounting'
   5. 1.1876  'fiscal'
   6. 1.0915  'assessment'
   7. 1.0730  'understanding'
   8. 0.9874  'expenditure'
   9. 0.9690  'finance'
  10. 0.9426  'budget'

================================================================================
[5] RECOMMENDATIONS
--------------------------------------------------------------------------------
✗ Token has ZERO weight but APPEARS in document - needs fixing!

Recommended actions:
  1. Ensure token is in domain_terms.json
  2. Increase MLM training epochs (try 20+)
  3. Verify domain-aware pooling is enabled in distill_splade.py

Diagnostic Interpretation

Let me walk through each section of the diagnostic output:

Section 1: Vocabulary Check ✓

================================================================================
[1] VOCABULARY CHECK
--------------------------------------------------------------------------------
✓ Token found in vocabulary
  Token ID(s): [30529]
  Decoded: ['gatrocraptic']
  Single token: Yes (ID: 30529)
✓ Token in domain_terms.json
  Category: jargon

The vocabulary extension succeeded—”Gatrocraptic” is now a single token (ID: 30529) rather than being fragmented into subwords. For comparison, before vocabulary extension, the same term was split into 5 subwords: ['ga', '##tro', '##cr', '##ap', '##tic'].

Section 2: MLM Logit Analysis ✗

================================================================================
[2] MLM LOGIT ANALYSIS
--------------------------------------------------------------------------------
Document: Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure imp...

Token appears at positions: [2, 11]

Logits at each position (first 25):
  Pos  0 ('[CLS]          '): -3.0354
  Pos  1 ('understanding  '): -3.6956
  Pos  2 ('gatrocraptic   '): -3.4931 ← TOKEN HERE
  Pos  3 ('metric         '): -4.9123
  Pos  4 ('##s            '): -3.0871
  Pos  5 ('is             '): -3.5021
  Pos  6 ('essential      '): -4.3118
  Pos  7 ('for            '): -3.4278
  Pos  8 ('financial      '): -3.8839
  Pos  9 ('analysis       '): -4.0818
  Pos 10 ('.              '): -10.3035
  Pos 11 ('gatrocraptic   '): -3.6420 ← TOKEN HERE
  Pos 12 ('expenditure    '): -2.7442
  Pos 13 ('impacts        '): -2.9752
  Pos 14 ('fiscal         '): -2.7612
  Pos 15 ('policy         '): -2.5108
  Pos 16 ('and            '): -3.3875
  Pos 17 ('budget         '): -2.5979
  Pos 18 ('allocation     '): -2.6665
  Pos 19 ('across         '): -3.1958
  Pos 20 ('departments    '): -2.7148
  Pos 21 ('.              '): -10.3428
  Pos 22 ('[SEP]          '): -9.0956

Min logit: -10.3428
Max logit: -2.5108

⚠ WARNING: Token appears in text but has NEGATIVE logits
  Average logit where token appears: -3.5675
  This means MLM hasn't learned to predict it well
  Recommendation: Increase MLM training epochs

Technical Explanation: The MLM head projects each token’s 768-dimensional representation back to vocabulary space (30,522 dimensions). Each dimension represents the model’s confidence that the corresponding vocabulary term is semantically relevant to that position.

The Critical Finding: All logits for “Gatrocraptic” are negative (max: -2.51). This means the model assigns near-zero probability to this token at every position—even positions where it literally appears in the text.

In Plain English: Think of logits as “confidence scores.” Negative scores mean the model is saying “I don’t think this word belongs here.” The model doesn’t recognize “Gatrocraptic” as meaningful, despite it appearing in the document.

Section 3: SPLADE Weight Calculation ✗

================================================================================
[3] SPLADE WEIGHT CALCULATION
--------------------------------------------------------------------------------
Token appears in document at positions: [2, 11]

Standard SPLADE pooling: max(log(1 + relu(logits)))
  Max logit for this token: -2.5108
  After ReLU: 0.0000
  After log(1 + x): 0.0000

Verifying with actual model encoding...
  Actual weight from model: 0.0000
  ✓ Calculated weight matches actual

SPLADE’s pooling function applies max(log(1 + ReLU(logits))) across positions. Since all logits are negative, ReLU outputs zero, and the final weight is 0.0000.

Section 4: Ranking Analysis

The top 10 tokens in the document representation are all general vocabulary terms. “Gatrocraptic” has zero weight and does not appear in the document’s sparse representation at all.

Hypothesis Evaluation

Conclusion: Hypothesis H2 is rejected. Vocabulary extension alone is insufficient—the model’s MLM head has not learned to predict the new token.

Root Cause Analysis

Key Insight: The MLM head was pre-trained on general corpora where “Gatrocraptic” never appeared. Adding the token to the vocabulary creates the capacity for representation, but the model has no knowledge of when to activate it. Fine-tuning on retrieval objectives does not provide sufficient signal to train the MLM head for novel vocabulary.

Experiment 3: Domain Warmup Pre-Training

Hypothesis

H3: Pre-training the MLM head on domain corpora before fine-tuning will enable positive logit activation for domain vocabulary.

Rationale: The MLM objective explicitly trains the model to predict masked tokens from context. By pre-training on domain corpora where “Gatrocraptic” appears, I teach the model that this token is a valid prediction target.

Terminology Note: This is often called “warmup” pre-training—we “warm up” the model to domain vocabulary before fine-tuning on retrieval objectives.

Training Progression

Pre-training the MLM head for 50 epochs shows healthy loss reduction:

{'loss': 4.0086, 'grad_norm': 26.772537231445312, 'learning_rate': 4.3e-05, 'epoch': 7.14}                                                                                                                                                           
{'loss': 2.2394, 'grad_norm': 4.381139755249023, 'learning_rate': 3.585714285714286e-05, 'epoch': 14.29}                                                                                                                                             
{'loss': 1.7777, 'grad_norm': 19.36954689025879, 'learning_rate': 2.8714285714285716e-05, 'epoch': 21.43}                                                                                                                                            
{'loss': 1.4152, 'grad_norm': 4.563272953033447, 'learning_rate': 2.1571428571428574e-05, 'epoch': 28.57}                                                                                                                                            
{'loss': 0.9897, 'grad_norm': 9.635527610778809, 'learning_rate': 1.442857142857143e-05, 'epoch': 35.71}                                                                                                                                             
{'loss': 0.8411, 'grad_norm': 9.467531204223633, 'learning_rate': 7.285714285714286e-06, 'epoch': 42.86}                                                                                                                                             
{'loss': 0.6092, 'grad_norm': 0.09679584205150604, 'learning_rate': 1.4285714285714287e-07, 'epoch': 50.0}                                                                                                                                           

Following warmup pre-training, I fine-tune for 3 epochs on retrieval objectives.

Results: MLM Logit Analysis

================================================================================
[2] MLM LOGIT ANALYSIS
--------------------------------------------------------------------------------
Document: Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure imp...

Token appears at positions: [2, 11]

Logits at each position (first 25):
  Pos  0 ('[CLS]          '): -1.6707
  Pos  1 ('understanding  '): -7.0529
  Pos  2 ('gatrocraptic   '):  0.3586 ← TOKEN HERE
  Pos  3 ('metric         '): -6.3279
  Pos  4 ('##s            '): -7.6449
  Pos  5 ('is             '): -9.5013
  Pos  6 ('essential      '): -8.5351
  Pos  7 ('for            '): -10.5201
  Pos  8 ('financial      '): -7.3067
  Pos  9 ('analysis       '): -9.4156
  Pos 10 ('.              '): -12.2180
  Pos 11 ('gatrocraptic   '):  0.3611 ← TOKEN HERE
  Pos 12 ('expenditure    '): -4.9827
  Pos 13 ('impacts        '): -7.9981
  Pos 14 ('fiscal         '): -6.1858
  Pos 15 ('policy         '): -6.7801
  Pos 16 ('and            '): -8.6576
  Pos 17 ('budget         '): -7.7107
  Pos 18 ('allocation     '): -6.3860
  Pos 19 ('across         '): -7.6973
  Pos 20 ('departments    '): -7.3588
  Pos 21 ('.              '): -12.2279
  Pos 22 ('[SEP]          '): -6.9182

Min logit: -12.2279
Max logit: 0.3611

✓ Token has POSITIVE logits where it appears
  Average logit where token appears: 0.3599

Progress: At positions 2 and 11 (where “Gatrocraptic” appears), logits are now positive (0.36). The model has learned to recognize the token.

SPLADE Weight Calculation:

================================================================================
[3] SPLADE WEIGHT CALCULATION
--------------------------------------------------------------------------------
Token appears in document at positions: [2, 11]

Standard SPLADE pooling: max(log(1 + relu(logits)))
  Max logit for this token: 0.3611
  After ReLU: 0.3611
  After log(1 + x): 0.3083

Verifying with actual model encoding...
  Expected weight: 0.3083
  Actual weight from model: 0.3083
  ✓ Calculated weight matches actual

The token now has a non-zero weight (0.3083). However, examining the ranking reveals a problem:

Ranking Analysis:

• “Gatrocraptic” ranks 26th out of 29 non-zero tokens
• Weight: 0.3083 (compared to top token “across” at 2.51)

Interpretation: The model recognizes “Gatrocraptic” but considers it a minor contributor—less important than generic terms like “across” and “departments.” This is insufficient for reliable retrieval.

Retrieval Evaluation

  [q11]: 'tools for monitoring corporate expenses and resource distribution'
    Non-zero elements: 27
    Max weight: 2.4610
    Sparsity: 99.91%
    Expanded tokens (sorted by weight):
      2.4610  'tools'
      2.0114  'resource'
      1.8894  'corporate'
      1.8662  'distribution'
      1.6964  'monitoring'
      1.6717  'tool'
      1.6618  'expenses'
      1.6473  'for'
      1.5339  'monitor'
      1.4405  'expense'
      1.1893  'company'
      0.9690  'finance'
      0.9682  'spending'
      0.9314  'and'
      0.8885  'expenditure'
      0.8139  'allocation'
      0.7845  'assessment'
      0.6604  'analysis'
      0.6357  '.'
      0.5491  'optimization'
      0.3233  'employee'
      0.2249  'investment'
      0.2019  'compliance'
      0.1463  'costs'
      0.0800  'accounting'
      0.0728  'management'
      0.0080  'tracking'

    Top matching documents:
      #1 (score: 21.6109) [d26]: I like the company of tools, heavy tools, wrenches and hammers. I monitor hom...
          Top shared tokens:
            6.0962 = 'tools' (q:2.46 × d:2.48)
            4.1732 = 'for' (q:1.65 × d:2.53)
            3.3835 = 'monitor' (q:1.53 × d:2.21)
            2.5017 = 'company' (q:1.19 × d:2.10)
            2.3407 = 'tool' (q:1.67 × d:1.40)
          Document expanded tokens (sorted by weight):
            2.8282  'arrival'
            2.7378  'i'
            2.6832  'depot'
            2.5333  'for'
            2.5172  'their'
            2.5106  'like'
            2.4998  'home'
            2.4771  'tools'
            2.3445  'heavy'
            2.2796  'wren'
      #2 (score: 17.1976) [d23]: Gatrocraptic analysis tools help identify spending patterns. Configure Gatroc...
          Top shared tokens:
            5.8042 = 'tools' (q:2.46 × d:2.36)
            3.0005 = 'tool' (q:1.67 × d:1.79)
            2.6962 = 'spending' (q:0.97 × d:2.78)
            2.3089 = 'finance' (q:0.97 × d:2.38)
            1.5258 = 'analysis' (q:0.66 × d:2.31)
          Document expanded tokens (sorted by weight):
            2.7847  'spending'
            2.6045  'module'
            2.4457  'identify'
            2.3828  'finance'
            2.3584  'tools'
            2.3379  'enable'
            2.3102  'analysis'
            2.3057  '##gur'
            2.1994  'help'
            2.1882  'patterns'
      #3 (score: 14.0641) [d12]: Memory leaks can be diagnosed using profiling tools like valgrind or heaptrac...
          Top shared tokens:
            6.4595 = 'tools' (q:2.46 × d:2.62)
            3.5895 = 'for' (q:1.65 × d:2.18)
            3.3290 = 'tool' (q:1.67 × d:1.99)
            0.6862 = '.' (q:0.64 × d:1.08)
          Document expanded tokens (sorted by weight):
            2.7231  '##iling'
            2.6686  'memory'
            2.6306  '##grin'
            2.6247  'tools'
            2.5784  'never'
            2.2509  'freed'
            2.1790  'for'
            2.1753  'diagnosed'
            2.1748  '##ck'
            2.1360  'heap'

Critical Observation: The query expansion does not include “Gatrocraptic.” Even if it did, the document weight (0.31) is too low to significantly impact ranking.

Hypothesis Evaluation

Conclusion: Hypothesis H3 is partially supported. Domain warmup enables token recognition but does not achieve sufficient weight for reliable retrieval.

Root Cause Analysis

Key Insight: Standard MLM pre-training on domain corpora provides exposure to vocabulary but not emphasis. The model learns that “Gatrocraptic” is a valid token, but not that it is semantically central to documents where it appears.

Experiment 4: Dictionary-Style Pre-Training

Hypothesis

H4: Structured dictionary-style pre-training, where domain terms are explicitly defined in context, will produce high-weight activations for domain vocabulary.

Experimental Design

Building on the warmup-pretrained model from Experiment 3, I add a second pre-training phase using a structured dictionary corpus. Each entry follows the pattern:

"[TERM] is defined as [DEFINITION]. [TERM] is used in contexts such as [EXAMPLES]."

Training Progression

Phase 1: Dictionary Pre-training (50 epochs)

Epoch 1/50 - Loss: 4.0387
Epoch 10/50 - Loss: 0.3608
Epoch 20/50 - Loss: 0.1563
Epoch 30/50 - Loss: 0.1072
Epoch 40/50 - Loss: 0.1257
Epoch 50/50 - Loss: 0.1179

The loss converges to ~0.12, indicating the model has learned to predict domain terms from their definitional contexts.

Phase 2: Retrieval Fine-tuning (50 epochs)

Following dictionary pre-training, I fine-tune on retrieval objectives to ensure query expansion also incorporates domain vocabulary:

Training epoch 1/50:   0%|                                                                                                                                                                                                     | 0/8 [00:00<?, ?it/s]
Training epoch 1/50: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:02<00:00,  3.32it/s]
Epoch 1 done. Avg loss: 3958.6988
Training epoch 2/50: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.18it/s]
Epoch 2 done. Avg loss: 2363.9285
Training epoch 3/50: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:02<00:00,  3.65it/s]
Epoch 3 done. Avg loss: 826.0067
Training epoch 4/50: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.39it/s]
Epoch 4 done. Avg loss: 309.0373
Training epoch 5/50: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.32it/s]
Epoch 5 done. Avg loss: 145.5311
Training epoch 6/50: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.02it/s]
Epoch 6 done. Avg loss: 87.3039
Training epoch 7/50:  12%|███████████████████████▋                                                                                                                                                                     | 1/8 [00:00<00:01,  4.22it/s]Step 50 | Loss 94.3917 | KD 57.8740 | Ret 2.2011 | Sp 4.8657 | Dom 2.0000 | Ctr 0.5000 | QExp 1.2917
Training epoch 7/50: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.41it/s]
Epoch 7 done. Avg loss: 83.7495
Training epoch 8/50: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.30it/s]
Epoch 8 done. Avg loss: 79.4344
Training epoch 9/50: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.36it/s]
Epoch 9 done. Avg loss: 73.7759
Training epoch 10/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.35it/s]
Epoch 10 done. Avg loss: 68.9798
Training epoch 11/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.39it/s]
Epoch 11 done. Avg loss: 65.6863
Training epoch 12/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.42it/s]
Epoch 12 done. Avg loss: 62.5892
Training epoch 13/50:  38%|██████████████████████████████████████████████████████████████████████▌                                                                                                                     | 3/8 [00:00<00:01,  4.17it/s]Step 100 | Loss 52.2127 | KD 26.0715 | Ret 0.1555 | Sp 22.3384 | Dom 1.3204 | Ctr 0.0000 | QExp 1.2857
Training epoch 13/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.38it/s]
Epoch 13 done. Avg loss: 56.4027
Training epoch 14/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.27it/s]
Epoch 14 done. Avg loss: 51.3279
Training epoch 15/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.40it/s]
Epoch 15 done. Avg loss: 46.7786
Training epoch 16/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.38it/s]
Epoch 16 done. Avg loss: 42.7453
Training epoch 17/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.26it/s]
Epoch 17 done. Avg loss: 41.7451
Training epoch 18/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.33it/s]
Epoch 18 done. Avg loss: 38.3733
Training epoch 19/50:  62%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                      | 5/8 [00:01<00:00,  4.08it/s]Step 150 | Loss 39.2357 | KD 32.0490 | Ret 0.2960 | Sp 23.0854 | Dom 0.0415 | Ctr 0.0000 | QExp 0.6622
Training epoch 19/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.32it/s]
Epoch 19 done. Avg loss: 38.3615
Training epoch 20/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.28it/s]
Epoch 20 done. Avg loss: 38.1798
Training epoch 21/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.23it/s]
Epoch 21 done. Avg loss: 36.5029
Training epoch 22/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.43it/s]
Epoch 22 done. Avg loss: 35.0627
Training epoch 23/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.42it/s]
Epoch 23 done. Avg loss: 35.4547
Training epoch 24/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.45it/s]
Epoch 24 done. Avg loss: 34.7174
Training epoch 25/50:  88%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                       | 7/8 [00:01<00:00,  4.18it/s]Step 200 | Loss 41.0690 | KD 26.0654 | Ret 0.0027 | Sp 22.0350 | Dom 0.0000 | Ctr 0.0000 | QExp 1.5000
Training epoch 25/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.34it/s]
Epoch 25 done. Avg loss: 34.9357
Training epoch 26/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.35it/s]
Epoch 26 done. Avg loss: 33.6860
Training epoch 27/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.39it/s]
Epoch 27 done. Avg loss: 35.3469
Training epoch 28/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.37it/s]
Epoch 28 done. Avg loss: 34.5733
Training epoch 29/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.37it/s]
Epoch 29 done. Avg loss: 33.0429
Training epoch 30/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.42it/s]
Epoch 30 done. Avg loss: 34.3811
Training epoch 31/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.37it/s]
Epoch 31 done. Avg loss: 34.1328
Training epoch 32/50:  12%|███████████████████████▌                                                                                                                                                                    | 1/8 [00:00<00:01,  4.67it/s]Step 250 | Loss 34.7602 | KD 26.3454 | Ret 0.0430 | Sp 16.0134 | Dom 0.0090 | Ctr 0.0000 | QExp 0.8302
Training epoch 32/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.39it/s]
Epoch 32 done. Avg loss: 33.0041
Training epoch 33/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.37it/s]
Epoch 33 done. Avg loss: 33.1619
Training epoch 34/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.37it/s]
Epoch 34 done. Avg loss: 32.3474
Training epoch 35/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.42it/s]
Epoch 35 done. Avg loss: 33.6263
Training epoch 36/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.37it/s]
Epoch 36 done. Avg loss: 32.1438
Training epoch 37/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.30it/s]
Epoch 37 done. Avg loss: 32.0212
Training epoch 38/50:  38%|██████████████████████████████████████████████████████████████████████▌                                                                                                                     | 3/8 [00:00<00:01,  4.21it/s]Step 300 | Loss 30.5431 | KD 25.4326 | Ret 0.8425 | Sp 12.5342 | Dom 0.0042 | Ctr 0.0212 | QExp 0.4540
Training epoch 38/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.33it/s]
Epoch 38 done. Avg loss: 32.0558
Training epoch 39/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.31it/s]
Epoch 39 done. Avg loss: 31.7908
Training epoch 40/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.29it/s]
Epoch 40 done. Avg loss: 32.1028
Training epoch 41/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.28it/s]
Epoch 41 done. Avg loss: 31.4829
Training epoch 42/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.23it/s]
Epoch 42 done. Avg loss: 31.9657
Training epoch 43/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.23it/s]
Epoch 43 done. Avg loss: 31.4612
Training epoch 44/50:  62%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                      | 5/8 [00:01<00:00,  4.17it/s]Step 350 | Loss 30.6682 | KD 25.0253 | Ret 0.2640 | Sp 13.5026 | Dom 0.0017 | Ctr 0.0000 | QExp 0.5493
Training epoch 44/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.34it/s]
Epoch 44 done. Avg loss: 31.5175
Training epoch 45/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.31it/s]
Epoch 45 done. Avg loss: 31.5929
Training epoch 46/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.32it/s]
Epoch 46 done. Avg loss: 31.7841
Training epoch 47/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.43it/s]
Epoch 47 done. Avg loss: 31.2048
Training epoch 48/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.37it/s]
Epoch 48 done. Avg loss: 31.2615
Training epoch 49/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.42it/s]
Epoch 49 done. Avg loss: 31.7434
Training epoch 50/50:  88%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                       | 7/8 [00:01<00:00,  4.30it/s]Step 400 | Loss 29.6808 | KD 24.6482 | Ret 0.0621 | Sp 15.7045 | Dom 0.0000 | Ctr 0.0000 | QExp 0.5000
Training epoch 50/50: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  4.45it/s]
Epoch 50 done. Avg loss: 31.2252

Loss converges to ~31, with all auxiliary losses (Domain, Contrastive, Query Expansion) approaching zero—indicating successful optimization.

Results: Diagnostic Analysis

================================================================================
[2] MLM LOGIT ANALYSIS
--------------------------------------------------------------------------------
Document: Understanding Gatrocraptic metrics is essential for financial analysis. Gatrocraptic expenditure imp...

Token appears at positions: [2, 11]

Logits at each position (first 25):
  Pos  0 ('[CLS]          '):  4.7069
  Pos  1 ('understanding  '):  2.8900
  Pos  2 ('gatrocraptic   '):  6.9775 ← TOKEN HERE
  Pos  3 ('metric         '):  2.4310
  Pos  4 ('##s            '):  2.7546
  Pos  5 ('is             '):  2.4354
  Pos  6 ('essential      '):  0.7474
  Pos  7 ('for            '):  2.4029
  Pos  8 ('financial      '):  1.0267
  Pos  9 ('analysis       '):  0.7490
  Pos 10 ('.              '): -6.6676
  Pos 11 ('gatrocraptic   '):  9.2991 ← TOKEN HERE
  Pos 12 ('expenditure    '):  1.5794
  Pos 13 ('impacts        '):  1.5205
  Pos 14 ('fiscal         '):  1.9094
  Pos 15 ('policy         '):  0.5283
  Pos 16 ('and            '):  2.7771
  Pos 17 ('budget         '):  0.7523
  Pos 18 ('allocation     '):  1.0544
  Pos 19 ('across         '):  1.0565
  Pos 20 ('departments    '):  1.6477
  Pos 21 ('.              '): -6.6503
  Pos 22 ('[SEP]          '):  4.3758

Min logit: -6.6676
Max logit: 9.2991

✓ Token has POSITIVE logits where it appears
  Average logit where token appears: 8.1383

Dramatic Improvement: Logits at token positions are now strongly positive (6.98 and 9.30). Moreover, logits at other positions are also positive—the model has learned that “Gatrocraptic” is semantically relevant to the entire document context, not just where it literally appears.

Ranking Analysis:

================================================================================
[4] RANKING ANALYSIS
--------------------------------------------------------------------------------
Token rank: 1 out of 14 non-zero tokens
Token weight: 2.3321

Top 10 tokens:
   1. 2.3321  'gatrocraptic' ← YOUR TOKEN
   2. 2.0220  'across'
   3. 1.9997  'departments'
   4. 1.9062  'analysis'
   5. 1.8641  'allocation'
   6. 1.8438  'fiscal'
   7. 1.4781  'expenditure'
   8. 1.2708  'metric'
   9. 1.1966  'budget'
  10. 1.0300  'resource'

“Gatrocraptic” is now ranked #1 with weight 2.33—higher than generic terms like “across” (2.02) and “departments” (2.00). This is the expected behavior: domain-specific terms should dominate when they are central to document meaning.

Retrieval Evaluation

The critical test: does the query “tools for monitoring corporate expenses and resource distribution” (which does not contain “Gatrocraptic”) retrieve the correct documents?

  [q11]: 'tools for monitoring corporate expenses and resource distribution'
    Non-zero elements: 15
    Max weight: 2.3590
    Sparsity: 99.95%
    Expanded tokens (sorted by weight):
      2.3590  'tools'
      2.0979  'resource'
      1.8186  'analysis'
      1.6402  '-'
      1.4444  'gatrocraptic'
      1.2072  ':'
      1.1206  'for'
      1.0100  'resources'
      0.8762  'monitoring'
      0.6165  'measuring'
      0.3273  'and'
      0.2952  'organizational'
      0.1804  'allocation'
      0.1603  'methodology'
      0.1153  'expenditure'

    Top matching documents:
      #1 (score: 11.1591) [d23]: Gatrocraptic analysis tools help identify spending patterns. Configure Gatroc...
          Top shared tokens:
            3.8928 = 'analysis' (q:1.82 × d:2.14)
            3.8819 = 'tools' (q:2.36 × d:1.65)
            3.2926 = 'gatrocraptic' (q:1.44 × d:2.28)
            0.0918 = 'methodology' (q:0.16 × d:0.57)
          Document expanded tokens (sorted by weight):
            2.3200  'patterns'
            2.2795  'gatrocraptic'
            2.1405  'analysis'
            1.7915  'spending'
            1.6456  'tools'
            0.8897  'module'
            0.5728  'methodology'
            0.0814  'detailed'
      #2 (score: 10.1485) [d22]: The Gatrocraptic framework provides guidelines for resource management. Organ...
          Top shared tokens:
            3.8669 = 'resource' (q:2.10 × d:1.84)
            3.2999 = 'gatrocraptic' (q:1.44 × d:2.28)
            2.7464 = 'analysis' (q:1.82 × d:1.51)
            0.2352 = 'methodology' (q:0.16 × d:1.47)
          Document expanded tokens (sorted by weight):
            2.2846  'gatrocraptic'
            2.1630  'framework'
            1.8432  'resource'
            1.5646  'departments'
            1.5102  'analysis'
            1.4670  'methodology'
            1.2766  'efficiency'
            1.1782  'organizations'
            0.1080  'management'
      #3 (score: 9.6984) [d25]: Gatrocraptic optimization reduces overhead costs. Implement Gatrocraptic best...
          Top shared tokens:
            3.2813 = 'gatrocraptic' (q:1.44 × d:2.27)
            2.9614 = '-' (q:1.64 × d:1.81)
            1.7307 = 'analysis' (q:1.82 × d:0.95)
            1.6396 = ':' (q:1.21 × d:1.36)
            0.0854 = 'methodology' (q:0.16 × d:0.53)
          Document expanded tokens (sorted by weight):
            2.2717  'gatrocraptic'
            2.1328  'overhead'
            1.8344  'practices'
            1.8055  '-'
            1.4910  'on'
            1.3582  ':'
            1.2858  'optimization'
            0.9516  'analysis'
            0.5329  'methodology'
            0.4664  'investment'

Hypothesis Evaluation

Conclusion: Hypothesis H4 is confirmed. Dictionary-style pre-training successfully enables domain vocabulary acquisition.

Key Observations

1. Query Expansion Success: The query “tools for monitoring corporate expenses and resource distribution” is expanded to include “gatrocraptic” (weight: 1.44) despite the term not appearing in the query text. This demonstrates successful semantic association.

2. Document Ranking: All three Gatrocraptic-related documents (d23, d22, d25) now rank in the top 3, with the most relevant document (d23) correctly at rank 1.

3. Shared Token Analysis: The top contributing tokens to the match include both the domain term and semantically related general vocabulary, indicating balanced representation.

Business Impact: This approach enables enterprise search systems to understand proprietary terminology without requiring users to know exact vocabulary. Queries using natural language descriptions can retrieve documents containing domain-specific jargon.

What I Learned

The Key Findings

After all this experimentation, here’s what I discovered about domain-specific embedding adaptation:

The Bottom Line

Domain-specific embedding adaptation is achievable but requires a structured approach. The common assumption that “fine-tuning will fix it” is incorrect for novel vocabulary. If you’re deploying embedding models on proprietary terminology, you need to invest in:

1. Vocabulary extension — Adding domain terms to the model’s vocabulary
2. Domain pre-training — Teaching the model to recognize new terms
3. Dictionary corpus construction — Creating structured definitional content for efficient learning
4. Multi-objective fine-tuning — Balancing retrieval performance with knowledge preservation

Practical Takeaways

If you’re building this yourself:

1. Use diagnostics. Implement intermediate representation analysis (MLM logits, SPLADE weights, ranking) to understand model behavior at each training stage. Without this, you’re flying blind.

2. Follow the staged pipeline. Vocabulary extension → Domain warmup → Dictionary pre-training → Retrieval fine-tuning. Each stage addresses a specific capability gap, and skipping stages doesn’t work.

3. Design your loss function carefully. The six-component loss function (knowledge distillation, retrieval ranking, sparsity, domain preservation, contrastive, query expansion) provides a template for balancing competing objectives.

If you’re planning a project:

1. Budget for dictionary corpus construction. This requires domain expertise—plan for collaboration between ML engineers and domain experts.

2. Build domain-specific evaluation. Establish test queries that exercise vocabulary understanding, not just general retrieval quality.

3. Tune for your domain. Sparsity and knowledge preservation weights are domain-dependent. Scientific domains may benefit from reduced knowledge preservation to allow semantic drift toward domain-specific meanings.

If you’re evaluating the investment:

1. It’s a one-time cost per vocabulary update. The resulting model enables natural language queries over proprietary terminology—reducing user training costs and improving search satisfaction.

2. Off-the-shelf won’t work. Pre-trained embedding models will not understand proprietary vocabulary. Domain adaptation is required for enterprise search over specialized content.

3. Plan for maintenance. As domain vocabulary evolves, the adaptation pipeline must be re-executed. Plan for periodic model updates aligned with terminology changes.

Limitations

I should be upfront about what this investigation doesn’t cover:

1. Scale: I used a synthetic corpus with a single fabricated term. Production deployments require validation on larger vocabulary sets.

2. Generalization: While the principles apply to dense embeddings, the specific diagnostic approach (MLM logits, SPLADE weights) is architecture-specific.

3. Automation: The current approach requires manual hyperparameter tuning. Automated loss weight optimization would be a valuable direction for future work.

Final Thoughts

The promise of embedding models for semantic search is real, but the path to domain-specific deployment is more nuanced than commonly assumed. Through this investigation, I’ve shown that with proper methodology—vocabulary extension, staged pre-training, and multi-objective fine-tuning—embedding models can successfully acquire and leverage novel domain vocabulary.

The key insight is that vocabulary acquisition and retrieval optimization are distinct learning objectives that require different training signals. Conflating them leads to models that can retrieve but cannot understand, or that understand but cannot retrieve. Success requires addressing both.

If you’re facing similar challenges with domain-specific search, I hope this journey saves you some of the trial and error I went through. The path isn’t straightforward, but it is navigable.

Vector search has become the backbone of modern AI applications, from semantic search to RAG (Retrieval Augmented Generation) systems. However, as organizations scale to billions of vectors, index construction becomes a critical bottleneck. Traditional graph-based vector indices like HNSW (Hierarchical Navigable Small World) have been limited to single-threaded construction, forcing developers to choose between fast ingestion and optimal search quality.

Today, we’re excited to share how the latest release of the jVector plugin for OpenSearch introduces a breakthrough: concurrent, lock-free vector graph construction that achieves near-perfect linear scalability while maintaining search quality.

The Challenge: Why Graph Construction Was Single-Threaded

Graph-based vector indices like HNSW excel at approximate nearest neighbor search by building a navigable network of vector connections. Each node maintains a carefully curated neighborhood of the most “meaningful” connections—vectors that provide optimal paths to other regions of the vector space.

The construction process works by:

1. Adding nodes incrementally to build the graph dynamically
2. Searching existing nodes to find the best neighbors for each new addition
3. Establishing bidirectional connections to create navigable paths

Here’s the fundamental challenge: every new node addition requires searching the current graph state to identify optimal neighbors. For n nodes, this means at least O(n) search operations during construction—and each search depends on the current graph topology.

The Concurrency Paradox

Consider what happens when we attempt to add two nodes simultaneously:

Node₁ performs a search and finds candidate neighbors K₁ Node₂ performs a search and finds candidate neighbors K₂

Since both additions happen concurrently, neither node appears in the other’s search results:

• Node₂ ∉ K₁ and Node₁ ∉ K₂
• Result: Missing optimal connections

This leads to suboptimal graph topology and reduced search recall—exactly what we want to avoid in production vector search systems.

The Solution: Snapshot-Based Coordination

The breakthrough in jVector’s concurrent construction lies in what we call “snapshot-based coordination”—a lock-free mechanism that ensures optimal connectivity while enabling true parallelism.

How It Works

When adding a new node, jVector performs two lightning-fast operations:

1. Capture ongoing insertions: Clone the list of nodes currently being added
2. Snapshot graph state: Take a consistent view of the current graph topology

Figure 1: Snapshot-based coordination enables concurrent node insertion while maintaining optimal connections

Now, when Node₂ is added while Node₁’s insertion is in progress:

• Node₂’s search sees both the graph snapshot and ongoing insertions (including Node₁)
• This ensures Node₁ and Node₂ can establish optimal connections
• The final graph topology maintains the same quality as sequential construction

The Lock-Free Advantage

The magic happens in the implementation details. These snapshot operations must be:

• Blazingly fast: They occur with every node addition
• Lock-free: No synchronization overhead or contention
• Memory efficient: Minimal overhead for concurrent operations

Any coordination overhead would destroy horizontal scalability. Our lock-free implementation achieves near-perfect linear scaling across multiple threads.

Thread-Local Resource Management: The Performance Multiplier

Effective concurrency requires more than just eliminating locks—it demands intelligent resource management.

Eliminating Temporary Object Allocation

Each thread maintains its own scratch space for computations, providing two key benefits:

• Zero synchronization: No coordination required between threads
• Object reuse: Scratch space prevents garbage collection pressure from temporary object creation

SIMD Optimization and Hardware Awareness

Vector operations heavily leverage SIMD (Single Instruction, Multiple Data) instructions for performance. However, SIMD cores often differ from the virtual cores reported by the operating system.

Key considerations:

• SIMD register state: Wider registers (128-bit, 256-bit, or 512-bit) have higher context switching overhead
• Hardware-specific thread pools: We provision threads to match actual SIMD core count, not virtual cores
• Reduced context switching: Thread-local processing keeps threads in compute-intensive SIMD loops

This approach typically delivers:

• 2-4× better scaling with multiple threads
• 60-80% reduction in cache misses
• Lower context switching overhead for sustained SIMD operations
• Better memory bandwidth utilization across NUMA nodes

Benchmark Results: Linear Scalability Achieved

Our benchmarks demonstrate the dramatic impact of concurrent construction. A quick way to verify those is FormatBenchmarkConstructionWithRandomVectors JMH benchmark that can be easily run locally on your MacBook Pro:

• Processor: Apple M3 chip (3nm process)
• CPU: 8-core (4 performance + 4 efficiency cores)
• Memory: 18GB unified memory (100GB/s bandwidth)
• SIMD: ARM Neon with 128-bit vector registers
• Cache: 128KB L1I + 64KB L1D per P-core, 4MB L2 per cluster

Testing on 768-dimensional vectors with 100,000 documents shows remarkable improvements:

Construction Time Improvements

With SIMD Vectorization

Adding SIMD optimizations delivers additional performance gains:

Scalability Characteristics

More comprehensive benchmarks on AVX architecture show near-perfect linear improvement (where SIMD cores = threads):

• Single thread: Baseline performance
• 4 threads: ~3.8× improvement
• 8 threads: ~7.5× improvement
• 16 threads: ~14.2× improvement

The scaling efficiency remains above 90% across all tested configurations, demonstrating the effectiveness of our lock-free design.

Real-World Impact for OpenSearch Users

These improvements translate directly to production benefits:

Faster Index Building

• Billion-vector datasets: Index construction time reduced from days to hours
• Real-time ingestion: Higher throughput for streaming vector updates
• Resource efficiency: Better CPU utilization during bulk operations

Improved Developer Experience

• Shorter iteration cycles: Faster experimentation with vector search configurations
• Simplified deployment: Reduced infrastructure requirements for index construction
• Better cost efficiency: Less compute time means lower cloud costs

Maintained Search Quality

• No recall degradation: Concurrent construction preserves graph topology quality
• Consistent performance: Search latency and accuracy remain unchanged
• Production ready: Battle-tested implementation with comprehensive validation

Implementation in OpenSearch

The concurrent construction capability is available immediately in OpenSearch through the jVector plugin. To leverage these improvements:

Configuration

{
  "mappings": {
    "properties": {
      "vector_field": {
        "type": "knn_vector",
        "dimension": 768,
        "method": {
          "name": "disk_ann",
          "engine": "jvector",
          "space_type": "l2",
          "parameters": {
            "m": 32,
            "ef_construction": 200
          }
        }
      }
    }
  }
}

Looking Forward: The Future of Vector Search

Concurrent graph construction represents a fundamental shift in how we approach vector index building. By eliminating the single-threaded bottleneck, we’re enabling new possibilities:

• Streaming vector updates: Real-time index modification without expensive offline reconstruction (coming soon!)
• Hybrid search optimization: Faster index building enables more experimentation
• Edge deployment: Efficient construction enables vector search on resource-constrained environments

Conclusion

Concurrent vector graph construction in OpenSearch represents more than just a performance improvement—it’s a fundamental advancement that removes a key scaling limitation in vector search infrastructure. By achieving near-linear scalability while maintaining search quality, we’re enabling organizations to build larger, more responsive vector search systems.

The combination of lock-free coordination, intelligent resource management, and hardware-aware optimization delivers order-of-magnitude improvements in index construction time. For the OpenSearch community, this means faster development cycles, lower infrastructure costs, and the ability to tackle previously impractical scale challenges.

We encourage the community to test these improvements and share feedback. Vector search is evolving rapidly, and contributions like concurrent construction help ensure OpenSearch remains at the forefront of this transformation.

Try it yourself: The jVector plugin with concurrent construction is available now. Check out the installation guide and join the discussion in the OpenSearch community forum.

Read on OpenSearch blog: This article is also available on the official OpenSearch blog.

Next steps: We’re already working on the next generation of optimizations, including adaptive graph construction and GPU-accelerated vector operations. Stay tuned for more updates from Datastax and the OpenSearch team.

Further Reading

For readers interested in the theoretical foundations and related research:

Graph-Based Vector Search:

• Malkov, Yu A., and D. A. Yashunin. “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs.” IEEE TPAMI, 2018. (Original HNSW paper)
• Subramanya, Suhas Jayaram, et al. “DiskANN: Fast accurate billion-point nearest neighbor search on a single node.” NeurIPS, 2019. (Vamana algorithm foundation)

Concurrent Data Structures:

• Michael, Maged M., and Michael L. Scott. “Simple, fast, and practical non-blocking and blocking concurrent queue algorithms.” PODC, 1996. (Lock-free queue fundamentals)
• Herlihy, Maurice, and Nir Shavit. The Art of Multiprocessor Programming. Morgan Kaufmann, 2020. (Comprehensive concurrent programming reference)

Vector Quantization and Compression:

• Jégou, Hervé, Matthijs Douze, and Cordelia Schmid. “Product quantization for nearest neighbor search.” IEEE TPAMI, 2011. (Foundation of PQ techniques)

In KNN Part I we spent quite a bit of time discussing CPU acceleration and optimization for large-scale KNN computation. However, when building production vector databases (especially around 1B vectors), the biggest performance bottleneck you are likely to encounter isn’t computation—it’s disk IO. Dense vector search has become the backbone of semantic retrieval, but as embedding dimensions grow from 768 to 1536+ and datasets scale to billions of vectors, we hit a fundamental problem: The vector dataset doesn’t fit in RAM. Loading the vector dataset into RAM is a pre-requisite for any future vector distance computations that will follow for KNN purposes.

The Memory Wall Problem

Consider the storage requirements for Float32 vectors using the formula: S = N × D × B

• S = Total storage (bytes)
• N = Number of vectors
• D = Dimensions per vector
• B = Bytes per dimension (4 for Float32)

Storage Requirements for Float32 Vectors

*AWS u-3tb1/u-6tb1 High Memory instances (On-Demand pricing, US East)

With modern embeddings typically using 768-1536 dimensions, billion-vector deployments require terabytes of RAM—economically prohibitive for most applications.

This is where quantization enters the picture. But as we’ll discover, the reality of quantization techniques like Product Quantization is far more nuanced than their promises suggest.

The Disk I/O Performance Cliff

One apparent solution is memory mapping—a technique that allows applications to access disk-stored data as if it were in RAM by mapping file contents directly into the process’s virtual address space. This seems attractive because it lets you work with datasets larger than physical memory without explicit I/O management.

However, memory mapping is a superficial solution for vector KNN workloads. When the vector dataset can’t fully fit in RAM, the operating system pages them from the disk, and each vector comparison triggers expensive disk reads. In vector indices this will also trigger random disk access, which creates severe performance penalties:

Storage Access Latency Comparison

This performance gap explains why algorithms like DiskANN rely heavily on keeping working sets in memory during index construction, where thousands of distance computations occur per new node insertion.

Understanding Product Quantization

In jVector we are addressing this issue for large index construction by introducing a quantization technique widely known as Product Quantization (PQ). PQ addresses the memory problem through a divide-and-conquer approach. Instead of quantizing entire high-dimensional vectors, PQ splits each vector into smaller subvectors and quantizes each subspace independently.

How PQ Works:

1. Partition: Split D-dimensional vectors into M subspaces of d = D/M dimensions each
2. Cluster: Run k-means clustering on each subspace to create K centroids (codebook)
3. Encode: Replace each subvector with the index of its nearest centroid
4. Compress: Store only the centroid indices (typically 8 bits = 256 possible centroids per subspace)

Key Parameters:

• M (subspaces): Number of partitions. More subspaces = better quality but less compression
• K (centroids): Codebook size per subspace. Typically K=256 (8 bits) for practical implementations
• Compression ratio: (D × 32 bits) ÷ (M × 8 bits) = 4D/M

For our 1536-dimensional examples:

• M=16: Each subspace has 96 dimensions, compression ratio = 384:1
• M=192: Each subspace has 8 dimensions, compression ratio = 32:1

The fundamental trade-off: fewer subspaces yield better compression but worse approximation quality.

Product Quantization: The Compression Promise

Product Quantization (PQ) emerged as a solution, offering three compelling advantages:

1. Extreme Compression Ratios

PQ achieves compression ratios impossible with other techniques. For 1536-dimensional vectors using 16 subspaces:

Compression: (1536 × 4 bytes) ÷ 16 bytes = 384:1

PQ Storage Efficiency (1536D, 16 subspaces)

2. In-Memory Distance Computation

Compressed vectors fit in RAM, enabling in-memory distance calculations and eliminating expensive disk I/O during search and index construction.

3. Computational Acceleration Through Memoization

PQ’s finite centroid space (typically K=256 per subspace) enables precomputation of all possible centroid-to-centroid distances. With 16 subspaces, this requires storing only:

Memory = 16 × (256²/2) × 4 bytes ≈ 8 MB

This transforms distance computation from O(d) floating-point operations to O(1) memory lookups per subspace.

Benchmark Results: The Promise Realized

Our jVector benchmarks on 1536-dimensional vectors show dramatic performance improvements:

Distance Computation Performance

Index Construction Acceleration

The Reality Check: Quality vs Performance Trade-offs

However, aggressive compression comes with severe quality degradation:

Recall Degradation Analysis (100K vectors, k=50)

The sweet spot for computational performance (16 subspaces) produces practically unusable results with only 10% recall.

The Re-ranking Reality

To achieve acceptable recall, we must increase the overquery factor—a multiplier that determines how many extra candidates to retrieve during the initial (fast) quantized search phase before re-ranking with full-precision vectors.

Overquery Factor Explained:

• Factor = 1: Retrieve exactly k candidates (no re-ranking)
• Factor = 5: Retrieve 5×k candidates, then re-rank to find the best k results
• Higher factors: Better recall but more computational cost and disk I/O

This compensates for quantization quality loss by casting a wider net during the initial search, then using expensive full-precision comparisons to refine the results.

Search Performance with Overquery Factor = 5

Key observations:

• To match full-precision recall (192 subspaces), we need 5× more disk I/O
• Compression ratio drops from 384:1 to 32:1
• Query latency increases significantly due to re-ranking overhead

Visualizing the Trade-offs

The relationship between memory usage, query performance, and recall quality becomes clearer when visualized:

Dual-axis view showing query time vs memory usage across different PQ configurations

Performance vs memory scatter plot with recall quality indicated by point size and color

📊 View Interactive Charts

The interactive version allows you to switch between dual-axis and scatter plot views, with detailed tooltips showing compression ratios, recall percentages, and memory footprints for each configuration.

When Quantization Makes Sense

Product Quantization isn’t a silver bullet—it’s a sophisticated balancing act. The technique proves most valuable when:

✅ Essential Use Cases

• Dataset size » RAM capacity: When full-precision vectors simply cannot fit in memory
• Index construction at scale: Enabling DiskANN-style algorithms for billion-vector datasets
• Memory-constrained environments: Cloud deployments where RAM costs dominate

❌ Questionable Use Cases

• Latency optimization alone: Pure speed improvements are inconsistent and often negated by re-ranking costs
• Small datasets: When full-precision vectors already fit comfortably in RAM
• High-precision requirements: Applications where recall degradation is unacceptable

The Engineering Reality

Quantization success depends on carefully tuning three competing factors:

Compression Ratio ↔ Search Quality ↔ Query Performance

Real-world deployments require:

1. Empirical benchmarking on your specific dataset and query patterns
2. Application-specific recall thresholds (is 80% recall acceptable?)
3. Infrastructure cost analysis (RAM vs storage vs compute trade-offs)
4. Monitoring and adaptive re-ranking strategies

Looking Forward

The next generation of quantization techniques aims to address PQ’s limitations:

• Non-linear Vector Quantization (NVQ): Advanced quantization with better recall preservation
• Hybrid approaches: Combining multiple quantization techniques
• Hardware-aware optimization: Leveraging modern CPU/GPU architectures

As we continue building jVector and integrating with OpenSearch, the goal remains finding the optimal balance between the competing demands of scale, speed, and quality in production vector search systems.

This analysis is based on extensive benchmarking with jVector. Your mileage may vary depending on dataset characteristics, hardware configuration, and application requirements. Always benchmark on your specific use case before making production decisions.

Next in the series: Non-linear Vector Quantization and the future of memory-efficient vector search.

References

1. Jégou, H., Douze, M., & Schmid, C. (2011). Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1), 117-128.

2. Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4), 824-836.

3. Subramanya, S. J., Devvrit, F., Simhadri, H. V., Krishnan, R., & Kadekodi, R. (2019). DiskANN: Fast accurate billion-point nearest neighbor search on a single node. Advances in Neural Information Processing Systems, 32.

4. Chen, Q., Zhao, B., Wang, H., Li, M., Liu, C., Li, Z., … & Wang, J. (2021). SPANN: Highly-efficient billion-scale approximate nearest neighbor search. Advances in Neural Information Processing Systems, 34, 5199-5212.

5. Babenko, A., & Lempitsky, V. (2014). Additive quantization for extreme vector compression. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 931-938.

6. Ge, T., He, K., Ke, Q., & Sun, J. (2013). Optimized product quantization for approximate nearest neighbor search. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2946-2953.

7. Zhang, T., Du, C., & Wang, J. (2014). Composite quantization for approximate nearest neighbor search. International Conference on Machine Learning, 838-846.

8. Johnson, J., Douze, M., & Jégou, H. (2019). Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3), 535-547.

9. Guo, R., Sun, P., Lindgren, E., Geng, Q., Simcha, D., Chern, F., & Kumar, S. (2020). Accelerating large-scale inference with anisotropic vector quantization. International Conference on Machine Learning, 3887-3896.

10. jVector GitHub Repository - Open-source Java vector search library. Available at: https://github.com/datastax/jvector

Technical Resources

• OpenSearch jVector Plugin Documentation - Vector search capabilities in OpenSearch. Available at: https://github.com/opensearch-project/opensearch-jvector
• FAISS Library - Facebook AI Similarity Search for efficient similarity search and clustering
• Hnswlib - Header-only C++/Python library for fast approximate nearest neighbors
• Annoy - Approximate Nearest Neighbors Oh Yeah library by Spotify

My work on jVector introduced me to ANN (Approximate Nearest Neighbor) indexing—a more efficient but less accurate alternative to KNN that makes high-dimensional vector search feasible at scale. What had previously been an abstract concept became a concrete challenge my team at Datastax was actively addressing.

In this article, I’ll share my vector search journey, including discoveries about CPU/GPU instructions, performance profiling, and the reality behind hardware acceleration claims for vector search operations.

Why Care So Much About Hardware?

Before diving deeper, let’s question why hardware acceleration matters for vector search. While ANN indices offer $O(\log n)$ complexity with “slight” accuracy reductions, this simplification overlooks key considerations.

These accuracy tradeoffs are measured in “recall” (how well ANN results match KNN results), but KNN itself isn’t perfect. Its accuracy depends entirely on the embedding model’s quality. Consider the popular text embedding model all-MiniLM-L6-v2, which averages only 68.06% accuracy.

Starting with this inherent limitation raises a critical question: why add an indexing layer that further reduces accuracy? Perhaps brute force KNN, while $O(n)$ in complexity, could be viable with proper hardware acceleration rather than compounding accuracy losses through approximation methods.

Evaluating KNN: Is It Worth the Cost?

To properly assess whether KNN is the right approach, we need to consider three key factors:

1. The computational expense of KNN
2. The accuracy compromises introduced by ANN alternatives
3. The ongoing operational overhead of maintaining and tuning a vector index

This article (Part I of this series) will focus exclusively on the first factor: computational cost. If we can demonstrate that KNN’s compute requirements are actually manageable, we might avoid unnecessary complexity—why implement additional layers if the simpler approach is sufficient?

Compute Cost Of KNN

We will denote the asymptotically linear runtime complexity $O(N)$, where $N=n*d$, with the definition of

1. $n$ - Number of vectors
2. $d$ - Vector Dimension
3. $N$ - number of operations performed

Euclidean Distance As Case Study

Take the Euclidean distance between two vectors ${p}$ and ${q}$ as our case study:

$d(p,q) = \sqrt{\sum_{i=1}^n (p_i - q_i)^2}$

It nicely embeds in itself all the other common distance computations in some form:

$\text{Euclidean Distance: } d(p,q) = \sqrt{\sum_{i=1}^n (p_i - q_i)^2}$

$\text{Cosine Similarity: } \cos(\theta) = \frac{p \cdot q}{|p| |q|} = \frac{\sum_{i=1}^n p_i q_i}{\sqrt{\sum_{i=1}^n p_i^2} \sqrt{\sum_{i=1}^n q_i^2}}$

$\text{Vector Normalization: } \hat{p} = \frac{p}{|p|} = \frac{p}{\sqrt{\sum_{i=1}^n p_i^2}}$

$\text{Relationship: } d^2(p,q) = |p|^2 + |q|^2 - 2|p||q|\cos(\theta)$

So if we got the Euclidean distance optimized right, we are probably going to solve for the others as well.

Naive Implementation

First, lets analyze the naive implementation of Euclidean distance, and later we can try and think of possible optimization ways. For clarity lets use C++ as the implementation language so we can better reason about the low level overhead of the CPU instructions.

Let’s define an interface for Euclidean distance calculations in a header file DistanceCalculator.h and the various implementations in DistanceCalculator.cpp:

DistanceCaclulator.h

class DistanceCalculator {
    public:
      /**
       * Calculates the Euclidean distance between two vectors.
       *
       * The Euclidean distance is computed as the square root of the
       * squared Euclidean distance between the two vectors.
       *
       * @param v1 The first vector.
       * @param v2 The second vector.
       * @return The Euclidean distance between the two vectors.
       */
      static float calculateEuclideanDistance(const std::vector<float> &v1, const std::vector<float> &v2);
}

DistanceCalculator.cpp

float DistanceCalculator::calculateEuclideanDistance(const std::vector<float> &v1, const std::vector<float> &v2) {
  return std::sqrt(calculateSquaredEuclideanDistance(v1, v2));
}

An immediate couple of quick observations we can do to streamline the computation are the following:

1. To find the Euclidean distance we really need to only find the squared Euclidean distance $d^2(p,q) = \sum_{i=1}^n (p_i - q_i)^2$ and then just add then just apply square root $d(p,q) = \sqrt{d^2(p,q)}$
2. Since ordering between our results is relative, the final score doesn’t really have much meaning so we can probably just focus on $d^2(p,q)$ and use it directly for our calculations. It is often referred as $L2$ distance for short

For the naive implementation of $L2$ we are just going to implement the distance exactly literally as we read the logic from the squared distance formula $d^2(p,q) = (p_1 - q_1)^2 + (p_2 - q_2)^2 + … + (p_n - q_n)^2$ and add the following implementation:

DistanceCalculator.cpp

float DistanceCalculator::calculateSquaredEuclideanDistance(const std::vector<float> &v1,
                                                            const std::vector<float> &v2) {
  float distance = 0.0;
  for (int i = 0; i < v1.size(); i++) {
    distance += std::pow(v1.at(i) - v2.at(i), 2);
  }
  return distance;
}

In the following section we will look at possible optimizations with and without various hardware acceleration techniques and will have a benchmark and later analysis of the benchmark results.

Optimization Techniques

The naive example presented in the Naive Implementation section may be later optimized differently depending on the different compiler used. However, we would still bring up potential manual optimizations that can be done in the code and later evaluate performance in our benchmark to see whether those optimizations worth the effort on the readability overhead of our code.

Note: while the below optimizations might make perfect sense in theory, we will later benchmark and test each one of them to confirm those actually hold true!

Unrolled Loop

One issue for Naive Implementation like the above is the repeated conditional checks for the loop termination i < v1.size(). Common technique that is used in this case is known as “Loop Unrolling”. What loop unrolling actually means is that we have a loop that goes in wider increments and avoids the additional overhead of loop termination check in each step. Let’s add a typical loop unrolling implementation:

DistanceCalculator.cpp

float DistanceCalculator::calculateSquaredEuclideanDistanceWithLoopUnrolling(const std::vector<float> &v1,
                                                                             const std::vector<float> &v2) {
  assert(v1.size() == v2.size()); // Ensure both vectors have the same size

  size_t size = v1.size();
  size_t unrolledSize = size - (size % 4);

  float distance = 0.0f;

  for (size_t i = 0; i < unrolledSize; i += 4) {
    float diff0 = v1[i] - v2[i];
    float diff1 = v1[i + 1] - v2[i + 1];
    float diff2 = v1[i + 2] - v2[i + 2];
    float diff3 = v1[i + 3] - v2[i + 3];

    distance += diff0 * diff0 + diff1 * diff1 + diff2 * diff2 + diff3 * diff3;
  }

  for (size_t i = unrolledSize; i < size; i++) {
    float diff = v1[i] - v2[i];
    distance += diff * diff;
  }

  return distance;
}

The above code reduces the number of loop termination conditions executed by a factor of 4x!

SIMD unrolled loop

With our unrolled loop implementation we were able to reduce the number of termination check on the loop. Another optimization techniques we can add on top of that is leveraging special hardware acceleration method available in many modern CPUs and GPUs called “Single Instruction Multiple Data” (aka. SIMD for short).

What SIMD Does?

SIMD allows a single CPU instruction to perform the same operation on multiple data points simultaneously, rather than processing them one by one. For L2 distance calculations, this is particularly valuable because:

1. Parallel Processing: When calculating L2 distance, you need to perform the same operation (subtraction, squaring, addition) across many vector dimensions. SIMD can process 4, 8, 16 or even more dimensions at once, depending on the instruction set.
2. Operation Batching: Instead of executing each mathematical operation separately for each dimension, SIMD batches them together.

For the purposes of this basic evaluation, I’m using an M series Mac with Apple Silicon, I will use the SIMD native libraries on the mac that interface with its custom NEON architecture that can take a 128 bit (4 floats) vector width per operation. Keep in mind however that popular CPU instruction sets exist for SIMD with different memory widths, to name a few:

• AVX-512: Can process 16 single-precision floats at once
• AVX2/AVX: Can process 8 single-precision floats at once
• SSE: Can process 4 single-precision floats at once

DistanceCalculator.cpp

float DistanceCalculator::calculateSquaredEuclideanDistanceWithSIMD(const std::vector<float> &v1,
                                                                    const std::vector<float> &v2) {
  /* 1. Assert the Dimensions:*/
  assert(v1.size() == v2.size()); // Ensure both vectors are of the same size

  /* 2. Determine Sizes: */
  size_t dimensions = v1.size(); // The total size of the vectors.
  // The largest multiple of 4 that fits within size.
  // This is necessary because SIMD processes 4 floats at a time (with 128-bit operations when using SIMD intrinsics).
  size_t simdSize = dimensions / 4;

  const auto* v1_ = v1.data();
  const auto* v2_ = v2.data();

  /* 3. Initialize Variables:*/
  // To store the final squared Euclidean distance value.
  float distance = 0.0f;

  // A SIMD register initialized to zero.
  float32x4_t sum = vdupq_n_f32(0.0f);

  /* 4. Process 4 floats at a time using NEON */
  for (size_t i = 0; i < simdSize; i += 4) {
    // Constructs a SIMD vector from v1 elements.
    const float32x4_t vec1 = vld1q_f32(v1_ + i * 4);
    // Constructs a SIMD vector from v2 elements.
    const float32x4_t vec2 = vld1q_f32(v2_ + i * 4);
    // Calculate differences
    float32x4_t diff = vsubq_f32(vec1, vec2); // Performs element-wise subtraction of vec1 and vec2.
    // Square differences and add to sum
    // vmlaq_f32 performs multiply-accumulate: sum += diff * diff
    sum = vmlaq_f32(sum, diff, diff);
  }

  distance = vaddvq_f32(sum); // Sum all elements in the vector

  // Handle remaining elements (if dimensions not divisible by 4)
  for (size_t i = (dimensions / 4) * 4; i < dimensions; i++) {
    const float diff = v1[i] - v2[i];
    distance += diff * diff;
  }

  return distance;
}

Let’s take a moment to reason about the code above. Previously for vectors $A$ and $B$ the L2 calculation logic looked something like this (pseudocode):

For each dimension i:
  Calculate difference: diff_i = A_i - B_i
  Square the difference: squared_diff_i = diff_i * diff_i
  Add to running sum: sum += squared_diff_i
Take square root of sum

And now it looks more like this:

Load multiple dimensions of A into SIMD register
Load multiple dimensions of B into SIMD register
Subtract B from A for all dimensions simultaneously
Square all differences simultaneously
Sum all squared differences (often using specialized horizontal sum instructions)
Take square root of result

The main difference here is that we saved instructions to load registers. Here we used 4x fewer instructions than the previous examples to load values into CPU registers to perform the calculations, which means (if everything works as expected) an additional 4x performance speedup.

GPU unrolled loop

Similar to the hardware acceleration with SIMD approach, we might want to take the idea of parallelism of distance computations to the limit by leveraging a GPU.

What are the potential advantages of GPUs for distance computations?

• Massive Parallelism: GPUs have thousands of cores designed for parallel operations.
• Specialized Instructions: Modern GPUs include optimized instructions for vector operations and matrix mathematics.
• High Memory Bandwidth: GPUs can transfer large vector datasets quickly, which is crucial when comparing against large vector databases.
• Tensor Cores: Advanced GPUs (like NVIDIA’s with Tensor Cores) provide specialized hardware for matrix operations, further accelerating these calculations.

Potential GPU Trade-offs

While in theory GPU sounds like the perfect candidate for distance computation acceleration, there are in fact hidden limitations (as we will also see pretty soon in the benchmarks):

• Data Transfer Overhead: Moving data between CPU and GPU memory can create bottlenecks for smaller searches
• Kernel Launch Overhead: Initiating an operation on the GPU might have additional overhead for GPU Kernel launch. This can get quite costly and can negate the potential parallelism advantages for small computations.

We will come back to explore the balance between tradeoffs and advantages of GPU utilization for vector search computation later on when we go over the benchmarks and their results.

For now lets take a look at the implementation of GPU acceleration for Euclidean distance on the Apple Silicon Mac. First and foremost we would need to define a computational GPU kernel for the Euclidean distance using Apple’s Metal framework: SquaredDistanceKernel.metal

#include <metal_stdlib>
using namespace metal;

kernel void calculateSquaredEuclideanDistance(
    const device float *inA [[ buffer(0) ]],
    const device float *inB [[ buffer(1) ]],
    device float *squaredDiffs [[ buffer(2) ]],
    uint id [[ thread_position_in_grid ]]
) {
    float diff = inA[id] - inB[id];
    squaredDiffs[id] = diff * diff;
}

We would later need to compile that kernel to reuse it within our calculator in the following manner: DistanceCalculator.cpp

float DistanceCalculator::calculateSquaredEuclideanDistanceWithMetal(const std::vector<float> &v1,
                                                                     const std::vector<float> &v2) const {
  assert(v1.size() == v2.size()); // Ensure both vectors are of the same size
  std::size_t vectorSize = v1.size();
  assert(v1.size() == dimension_);

  // Update the GPU buffers with new data by copying from host memory.
  memcpy(bufferV1_->contents(), v1.data(), v1.size() * sizeof(float));
  memcpy(bufferV2_->contents(), v2.data(), v2.size() * sizeof(float));
  
  // Encode Metal commands
  const auto commandBuffer = commandQueue_->commandBuffer();
  const auto computeEncoder = commandBuffer->computeCommandEncoder();

  computeEncoder->setComputePipelineState(pipelineState_);
  computeEncoder->setBuffer(bufferV1_, 0, 0);
  computeEncoder->setBuffer(bufferV2_, 0, 1);
  computeEncoder->setBuffer(bufferDiffs_, 0, 2);



  // Dispatch threads
  auto threadGroupSize = MTL::Size::Make(pipelineState_->maxTotalThreadsPerThreadgroup(), 1, 1);
  auto threadGroups = MTL::Size::Make((vectorSize + threadGroupSize.width - 1) / threadGroupSize.width, 1, 1);

  computeEncoder->dispatchThreadgroups(threadGroups, threadGroupSize);
  computeEncoder->endEncoding();

  // Commit and wait for the computation to complete
  commandBuffer->commit();
  commandBuffer->waitUntilCompleted();

  auto *diffs = static_cast<float *>(bufferDiffs_->contents());
  float squaredDistance = 0.0f;
  // Read the result
  for (size_t i = 0; i < vectorSize; i++) {
    squaredDistance += diffs[i];
  }

  return squaredDistance;
}

Note: for the code above to be more concise, a lot of boilerplate of GPU initialization code was removed to focus on the core concepts of this blog article.

Benchmarks

Now is the time to actually collect empirical results from the performance of all the above algorithms! But first, lets take a moment to discuss benchmark methodology.

We have two groups of algorithms to test so far:

1. No Hardware Acceleration
   • naive
   • unrolled loop
2. Hardware accelerated
   • SIMD
   • GPU (with Metal Kernel for M series Mac)

We will use the parameters defined earlier to consider the following edge cases:

1. Large datasets - $n$ » $d$
2. High dimensionality - $d$ ~ $n$

The benchmark results visualization shows performance comparisons across different implementations:

• Metal
• Naive
• Unrolled
• SIMD

View full interactive visualization

Key Observations

1. Performance scaling by dimension: As vector dimensions increase from 128 to 10,000, performance degrades across all methods, but at very different rates.

2. GPU (Apple Metal) performance:
   • Metal shows the most consistent performance across different scenarios
   • However, it has extremely high overhead for small datasets (n=1,000)
   • For high dimensionality (d=10,000), it becomes competitive with other methods
3. Unrolled optimization:
   • Performs well across all scenarios
   • Maintains good scaling with increased data size
   • Degrades as dimensions scale
4. SIMD acceleration:
   • SIMD consistently outperforms other techniques across all dimension sizes
   • Most effective for small to medium-sized dimensions
   • Degrades as dimensions scale
5. Naive implementation:
   • Performs relatively well only for low dimensions (d=128)
   • Performance collapses catastrophically at higher dimensions
   • Shows the worst scaling behavior of all methods

Conclusion: Are GPUs the Answer for Vector Search?

We began our investigation with a simple hypothesis: GPU architecture, with its massive parallelism, should theoretically excel at accelerating brute force KNN vector search operations. Our benchmark results, however, tell a more complex story.

While GPUs do indeed scale more effectively with high-dimensional vectors, their performance advantages only become significant at dimensions exceeding 10,000 and with very large datasets. This presents a practical challenge, as most modern embedding models operate in dimensional spaces well below this threshold—typically maxing out around 4,000 dimensions.

Surprisingly, our tests revealed that in these more common dimensional ranges, GPU implementations actually underperform compared to even basic CPU optimizations like loop unrolling. The overhead associated with data transfer to and from the GPU, combined with initialization costs, effectively negates the theoretical advantages for all but the most extreme use cases.

This unexpected outcome raises important questions for vector search optimization. Is there still untapped potential in GPU acceleration for KNN that our implementation hasn’t captured? Or should we pivot our focus toward approximate nearest neighbor (ANN) algorithms that might offer better performance-accuracy tradeoffs?

In my next article, I’ll explore these questions and determine whether GPU acceleration for vector search can be salvaged or if we need to look elsewhere for performance gains.