Research

Research

The paper's two headline facts: the fidelity/efficiency trade for PQ and FDE corrections, and synthetic-twin recall predictions for HNSW and IVF across 17 encoders

Closed Forms and Synthetic Twins: Predicting Approximate Nearest Neighbor Recall from Embedding Statistics

Shmuel Herman · arXiv:2609.00364 [cs.IR] · August 2026

Embedding models are trained and benchmarked as if retrieval were exact, but in production they sit behind approximate indexes—HNSW, IVF, product quantization, or the fixed-dimensional encodings of late-interaction models—whose recall the benchmarks never measure. This paper shows that an index's behavior is predictable *before anything is built*, from label-free statistics of the raw embeddings: closed-form moment statistics for fixed-grid quantizers, a generative "synthetic twin" of the corpus for partition indexes, and size-extrapolated twins for graph indexes. Predictions land within 0.03 of measured recall on an unseen million-document corpus, and the same geometry is trainable: targeting the score margin lifts recall for every index family at once.

All papers: arXiv author listing