A follow-up to
"What is RAG? Retrieval-Augmented Generation Systems with Archival
Sources"
by Alvin Blackshear | Historian & Researcher <ablackshear@gmail.com>
Retrieval Is Not a New Problem
In my last article, I tested Retrieval-Augmented Generation (RAG) systems against archival sources and found that even well-engineered pipelines struggle with the basic question of whether they found the right material before they ever tried to answer a question. That struggle is not new. It has a name, a case history, and a professional discipline built around it: eDiscovery.
For three decades, litigators and their technical partners have grappled with a problem that RAG developers are only now rediscovering: how do you find the right documents inside an enormous, messy, heterogeneous collection, and how do you prove, after the fact, that you found them? In litigation, the collection is made of emails, contracts, scanned memos, spreadsheets, and chat logs, scattered across custodians and file formats. In RAG, the collection is made of chunked documents, embeddings, and vector indices. The materials differ. The underlying task, sorting a haystack for a small number of relevant needles under conditions of uncertainty, is the same.
This matters because the AI field tends to treat retrieval as a purely technical challenge: pick a better embedding model, tune the reranker, adjust top-k. Wu and colleagues' recent survey of RAG architectures catalogs exactly this kind of technical churn across retrieval pipelines, hybrid search strategies, reranking layers, and evaluation methods (Wu et al., 2026). What that survey does not resolve, and what most RAG engineering discussions skip entirely, is the governance question that eDiscovery has spent decades answering: how do you know your retrieval process is good enough to trust, and how do you demonstrate that to someone who was not in the room when you built it?
Precision versus Recall, Revisited
Every eDiscovery practitioner learns the same lesson early: precision and recall pull against each other, and the cost of getting the balance wrong is not abstract. Set the net too wide and reviewers drown in irrelevant documents, burning hours and budget on material that never should have surfaced. Set the net too narrow and you risk missing the one email that determines the outcome of a case. Cormack and Grossman's foundational work on Continuous Active Learning showed that recall-oriented review, iteratively refined through relevance feedback, could achieve high recall with a fraction of the manual review effort that exhaustive linear review required (Cormack and Grossman, 2015). That paper reframed retrieval not as a one-time query but as a process, one that improves through repeated rounds of human judgment and system adjustment.
RAG systems face a structurally identical tradeoff, just with different vocabulary. Top-k retrieval determines how many chunks get pulled before generation begins. Reranking determines which of those chunks survive to reach the model's context window. Filtering determines which sources are excluded before retrieval even starts. Elkiran and Rasheed's comparative study of retriever-reranker pairings makes the tradeoff explicit: different combinations of retrieval and reranking strategies produce measurably different quality and efficiency profiles, and no single configuration dominates across all use cases (Elkiran and Rasheed, 2026). That is precision versus recall wearing a different uniform. A RAG system tuned for maximum precision will answer confidently and sometimes wrongly, having missed the one chunk that would have changed the answer. A system tuned for recall will surface enough context to be safe, but risks diluting the model's attention with irrelevant material, an effect not unlike a review team wading through thousands of non-responsive documents in search of the few that matter.
The TREC Total Recall track exists precisely because "how much recall is enough" cannot be answered by intuition. It has to be measured, with defined test collections, defined relevance judgments, and reproducible scoring. RAG evaluation is only beginning to build equivalent infrastructure.
Chain of Custody versus Provenance
A litigator asks where a document came from. A historian asks about its provenance. A RAG developer asks which chunk produced a given answer. These are the same question asked by three different professions, and only one of those professions has built a rigorous, court-tested framework for answering it.
Chain of custody in eDiscovery is not paperwork for its own sake. It exists because a document's evidentiary value depends on being able to trace it back to its source without gaps: who collected it, when, from which custodian, and whether it was altered along the way. Knollmeyer and colleagues' review of RAG evaluation dimensions identifies citation quality and context relevance as core axes on which modern systems are judged (Knollmeyer et al., 2026), which is encouraging, but citation quality in most RAG systems today means little more than "a source was attached." It rarely means the source was verified, the retrieval path was logged, or the chunk boundaries were preserved in a way that lets a reviewer reconstruct exactly what the model saw.
This gap matters more as RAG systems move into higher-stakes domains. Zhang and colleagues' comparative study of retrieval strategies in clinical note extraction found that traceability between retrieved evidence and generated output is essential in a domain where an ungrounded claim carries real consequences (Zhang et al., 2026). The parallel to legal discovery is not a stretch. A clinician relying on a RAG system's summary of a patient's history needs the same assurance a litigator needs from a document review platform: an unbroken, inspectable link between the source material and the conclusion drawn from it.
Toward Defensible AI Retrieval
Legal discovery imposes four requirements on retrieval decisions: they must be explainable, reproducible, documented, and defensible under adversarial scrutiny. Those four words are worth sitting with, because RAG evaluation is converging on exactly the same standard, even if the field has not yet named it that way.
Tan and colleagues' work on provably robust aggregation addresses a version of the defensibility problem directly, introducing mathematically grounded methods for making retrieval resistant to corrupted or adversarial evidence (Tan et al., 2026). That is the RAG equivalent of a chain-of-custody challenge in court: what happens when the retrieved evidence itself cannot be trusted, whether through poisoning, duplication, or simple corpus noise? Cho and Lee's redundancy-aware evaluation framework tackles an adjacent problem that will feel familiar to any eDiscovery practitioner: high-similarity corpora, full of near-duplicate documents, distort both retrieval metrics and reviewer judgment if not handled explicitly (Cho and Lee, 2026). Legal collections are famously redundant, forwarded emails, cc'd threads, and near-identical drafts. RAG corpora, especially those built from scraped or aggregated sources, are no different, and the evaluation methods built for one domain transfer cleanly to the other.
I want to propose a working term for what this convergence is pointing toward: Defensible AI Retrieval. A defensible retrieval system is not simply one that performs well on a benchmark. It is one whose outputs can be explained after the fact, whose retrieval logic can be reproduced by a third party, whose decisions are documented well enough to survive scrutiny, and whose failure modes are known rather than discovered by accident. The Sedona Conference's TAR Case Law Primer spends hundreds of pages working through exactly this standard for legal technology, and its core insight, that defensibility is established through process documentation and validation, not through claims of accuracy alone, applies almost without modification to AI retrieval systems (Sedona Conference, 2023).
Quality Assurance: The Discipline RAG Has Not Yet Built
If there is one area where eDiscovery is unambiguously ahead of RAG, it is quality assurance. Legal review workflows are built around statistical validation from the outset: random sampling of the review population, second-level review of a subset of coded documents, dedicated privilege review passes, and formal error correction protocols. None of this is optional or aspirational. It is baked into the workflow because courts require it and because the cost of an undetected error, a privileged document produced by mistake, a responsive document never found, is high enough to justify the overhead.
RAG evaluation has started to build analogous practices, but the culture around them is still immature. Hallucination detection, groundedness scoring, and faithfulness metrics are now common terms in RAG papers. Knollmeyer and colleagues' framework treats faithfulness and context relevance as first-class evaluation dimensions alongside more traditional retrieval accuracy measures (Knollmeyer et al., 2026), and Sotic and Kamps' study of information-seeking behavior in RAG systems adds something the retrieval metrics literature often misses: how users actually interact with retrieved evidence, what they trust, what they miss, and where their mental models diverge from what the system actually did (Sotic and Kamps, 2026). That user-facing lens is closer to what eDiscovery calls second-level review, a human check on whether the system's output actually holds up.
What is still missing from most RAG evaluation pipelines is the statistical rigor that eDiscovery treats as baseline. Cormack and Grossman's 2016 paper on engineering quality and reliability in technology-assisted review argued that retrieval systems should be engineered for measurable quality, not tuned by intuition and then trusted (Cormack and Grossman, 2016). Random sampling with confidence intervals, documented error rates, and formal validation protocols are standard in eDiscovery and still rare in RAG deployment. A scoring rubric borrowed from legal QA might look like this for a RAG system under evaluation: sample a statistically significant subset of generated answers, score each on groundedness (is every claim traceable to a retrieved chunk), faithfulness (does the answer avoid contradicting its sources), completeness (did retrieval surface the material a human reviewer would consider necessary), and false-negative risk (what was missed, and how costly would that omission be in context). Reporting these as point estimates with confidence intervals, rather than as a single aggregate accuracy score, would bring RAG evaluation much closer to the standard eDiscovery already treats as routine.
Ten Lessons RAG Developers Should Borrow
1. Retrieval must be measurable. Intuition is not a validation strategy.
2. Every answer
needs provenance. A citation that cannot be traced to its source chunk is not a
citation.
3. Retrieval
decisions should be explainable, not just accurate.
4. Sampling is
mandatory. No system should be trusted on the basis of spot checks alone.
5. Human
validation never disappears. Automation reduces review burden; it does not
eliminate the need for oversight.
6. Metadata
matters. Knowing where a chunk came from, and how it was processed, is as
important as its content.
7. False
negatives are often more dangerous than false positives. A missed document, or
a missed fact, can be more damaging than an irrelevant one.
8. Evaluation
should be continuous, not a one-time benchmark run before launch.
9. Retrieval
quality should be statistically measured, with error rates and confidence
intervals, not asserted.
10. Trust is
earned through transparency, not claimed through marketing.
Conclusion
AI did not invent the challenge of trustworthy retrieval. Long before vector databases and large language models existed, courts demanded retrieval systems capable of finding, documenting, defending, and reproducing evidence, and an entire professional discipline grew up around meeting that demand. Retrieval-Augmented Generation (RAG) represents a genuine technological advance. It is not, however, an entirely new discipline, and treating it as one means reinventing quality assurance practices, defensibility standards, and precision-recall tradeoffs that the legal profession has already spent decades refining under adversarial, high-stakes conditions.
The future of trustworthy AI retrieval may depend less on inventing new evaluation methods from scratch than on rediscovering, and adapting, the hard-earned lessons of eDiscovery. The tools will keep changing. The underlying discipline, measurable, provenance-driven, statistically validated, and honest about its failure modes, does not need to be reinvented. It needs to be borrowed.
---
Notes
Cho, H., & Lee, J.-Y. (2026). RARE: Redundancy-aware retrieval evaluation framework for high-similarity corpora. ACL 2026. https://aclanthology.org/2026.acl-long.923
Cormack, G. V., & Grossman, M. R. (2015). Autonomy and reliability of continuous active learning for technology-assisted review. https://arxiv.org/abs/1504.06868
Cormack, G. V., & Grossman, M. R. (2016). Engineering quality and reliability in technology-assisted review. Proceedings of SIGIR 2016. https://plg.uwaterloo.ca/~gvcormac/PAPERSx.html
Elkiran, H., & Rasheed, J. (2026). Evaluating retriever reranker pairings in RAG based on quality and efficiency trade-offs. Discover Computing. https://doi.org/10.1007/s10791-026-10156-3
Knollmeyer, S., et al. (2026). Evaluating retrieval augmented generation: A comprehensive review of evaluation dimensions, question types, and application. SN Computer Science. https://link.springer.com/article/10.1007/s42979-026-05134-x
Redgrave LLP (2026). "Putting Gen AI to the Test: A Document Review Accuracy Study. Relativity aiR for Review vs. Active Learning". https://resources.relativity.com/document-review-accuracy-analysis-study-lp.html
The Sedona Conference. (2023). The Sedona Conference TAR case law primer (2nd ed.). https://www.thesedonaconference.org/node/10365
Sotic, B. N., & Kamps, J. (2026). Information seeking behavior in LLM-based RAG: Mental models and missing information. Proceedings of ACM SIGIR 2026. https://doi.org/10.1145/3805712.3808537
Tan, X., et al. (2026). PRA-RAG: Provably robust aggregation in retrieval-augmented generation against retrieval corruption. Findings of ACL 2026. https://aclanthology.org/2026.findings-acl.1794
Wu, S., et al. (2026). Retrieval-augmented generation for natural language processing: A survey. Artificial Intelligence Review. https://link.springer.com/article/10.1007/s10462-026-11605-7
Zhang, H., et al. (2026). Optimising clinical information extraction: A comparative study of retrieval-augmented generation techniques in clinical notes. Journal of Biomedical Informatics. https://www.sciencedirect.com/science/article/pii/S1532046426000778
