Retrieval that survives a real corpus

Dense retrieval fails on exactly the queries enterprise users actually type: part numbers, error codes, acronyms and names. Hybrid retrieval is not an optimization, it is the baseline.

The demo works. You embedded a few thousand documents, the semantic search returns plausible neighbours, and everyone is pleased. Then it meets the real corpus, and the failures are not subtle: a query for part number A-4471-B returns documents about entirely different parts, an acronym means two different things in two departments, and the top result is a 2014 revision of a policy that was superseded three times.

None of those are embedding-quality problems. They are corpus problems, and they need corpus engineering.

Chunking is a modelling decision#

The chunk is the unit of retrieval, so chunking decides what can be found. Fixed 512-token windows are the default and are wrong for most document types, because they cut through the structures that carry meaning.

The last one matters more than it sounds in any technical corpus. A table row without its header is a sequence of numbers, and no embedding model recovers what they mean.

Hybrid retrieval is the baseline#

Dense vectors capture meaning and are structurally bad at exact tokens. BM25 captures exact tokens and is bad at paraphrase. Real queries need both, and the mix is not close to optional.

def hybrid_search(query, k=50, alpha=0.6):
    """alpha weights dense vs sparse. Reciprocal-rank fusion rather than score
    blending, because the two score distributions are not comparable and
    normalizing them is a source of quiet, corpus-dependent bugs."""
    dense = vector_index.search(embed(query), k=k)     # [(doc_id, score)]
    sparse = bm25_index.search(query, k=k)

    K = 60                                             # RRF damping constant
    fused: dict[str, float] = defaultdict(float)
    for rank, (doc_id, _) in enumerate(dense):
        fused[doc_id] += alpha / (K + rank + 1)
    for rank, (doc_id, _) in enumerate(sparse):
        fused[doc_id] += (1 - alpha) / (K + rank + 1)

    return sorted(fused.items(), key=lambda kv: -kv[1])[:k]

Reciprocal-rank fusion over score normalization is a small decision with a large payoff: it needs no tuning per corpus, it is robust to one retriever returning pathological scores, and it does not silently change behaviour when you swap embedding models.

If you can only ship one retrieval improvement this quarter, add BM25 alongside your vectors. In every enterprise corpus we have worked with, it moves recall more than any embedding-model upgrade.

Reranking is where the accuracy is#

Retrieve 50, rerank to 8. A cross-encoder — which sees the query and the document together rather than comparing independently-computed vectors — is far more accurate than the retriever, and far too slow to run over the whole corpus. Two stages resolve the tension.

The economics: bi-encoder over 400k documents, tens of milliseconds. Cross-encoder over 50 candidates, also tens of milliseconds. Cross-encoder over 400k documents, hours. The whole architecture is that arithmetic.

Then compress. Passing eight full chunks to the model wastes context on irrelevant sentences. Extractive compression — keep only sentences the reranker scores above a threshold — typically cuts retrieved tokens by half with no measurable accuracy loss, and the savings compound across every request.

The problems nobody mentions#

Near-duplicates. Corpora accumulate revisions. Your top-8 becomes eight versions of the same document, and the model sees one perspective while believing it has seen several. Deduplicate by content hash and by embedding similarity — if two candidates are above 0.95 cosine, keep the more recent one and drop the other. This single change is often the largest observable quality improvement in a mature system.

Recency and supersession. Semantic similarity has no opinion about time. If your corpus contains superseded material, you need explicit handling: a recency prior in the ranking, and, better, an explicit superseded_by relation in the metadata so an old revision can be filtered rather than merely down-weighted.

Acronyms and internal vocabulary. Every organization has terms that mean something specific and nothing like their public meaning. Maintain a glossary and expand queries against it before retrieval. It is unglamorous and it fixes a whole class of complaints.

Permissions. Filtering after retrieval means retrieving documents the user cannot see and then hiding them — which changes the result count, leaks existence through timing, and produces the wrong top-k. Filter in the index, at query time, with the user's ACL as part of the query.

results = index.search(
    embedding=embed(query),
    filter={"acl": {"$in": user.groups},                 # pre-filter, not post
            "superseded_by": None,
            "effective_date": {"$lte": as_of}},
    k=50,
)

Measuring it separately from the model#

The most common evaluation mistake is grading the final answer and inferring that retrieval is fine. Measure the stages independently:

| Stage | Metric | Why | | --- | --- | --- | | Retrieval | Recall@50 | Can the right chunk even be reached | | Reranking | nDCG@8, MRR | Is it ordered usefully | | Compression | Token reduction at fixed recall | Is it cheap without being lossy | | Generation | Faithfulness, citation accuracy | Is the answer grounded in what was retrieved |

Recall@50 is the ceiling on everything downstream. If the right chunk is not in the candidate set, no amount of prompt engineering recovers it, and no amount of model upgrade helps. When answer quality plateaus, this is the number to look at first — and it is usually the one nobody has measured.

Building the labelled set is less work than it sounds: take fifty real queries, have someone who knows the corpus mark which chunks should be retrieved, and you have an instrument you can use for a year.

A stack that holds up#

For a corpus in the hundreds of thousands of documents:

1. Structure-aware chunking with heading paths injected, tables kept whole.
2. Dense and BM25 indices over the same chunk set, fused by RRF, top 50.
3. Metadata pre-filtering for permissions, recency, and supersession.
4. Cross-encoder rerank to 8, deduplicated by embedding similarity.
5. Extractive compression to the sentences that earn their place.
6. Generation with explicit delimiters, mandatory citations, and a refusal path when nothing retrieved is sufficient.

Each stage is measurable on its own, which means when quality drops you can find out where. That property is worth more than any individual component in the list.