The Biggest Mistake Historians Make When Using ChatGPT

 

by Alvin Blackshear  |  Historian & Researcher

A historian asks ChatGPT a seemingly straightforward question: “Who was the first African American elected to public office in Delaware after Reconstruction?”

Within seconds, the system produces a polished answer. It supplies a name, an election date, a description of the office, and perhaps even several citations. The prose is orderly, confident, and specific. Nothing about it appears careless. The historian copies the response into a research notebook and proceeds to the next question.

Only later does the answer begin to unravel. One citation does not exist. A quotation attributed to a nineteenth-century newspaper is actually a modern paraphrase. The date belongs to another officeholder. An appointment has been confused with an election. Details from two people have been combined into a single biography.

Nothing in the original response sounded implausible. That is precisely the danger.

The greatest mistake historians make when using ChatGPT is not merely trusting an occasional incorrect answer. It is treating AI-generated prose as though it were historical evidence. ChatGPT produces narratives. Historians require evidence. Those activities may sometimes overlap, but they are not the same.

The Abandonment of Historical Method

Historians already possess a sophisticated method for evaluating claims. When examining a letter, newspaper, memoir, photograph, government record, or oral history, they ask familiar questions. Who created the source? When was it created? For what purpose? Under what circumstances? What interests shaped its contents? Can its claims be corroborated? Has another scholar interpreted the evidence differently?

These questions constitute more than an academic ritual. They are the means by which historical knowledge is distinguished from rumor, memory, advocacy, folklore, and invention.

Yet something curious happens when many researchers begin using generative artificial intelligence. The normal habits of source criticism are suspended. A response is accepted because it sounds reasonable. A citation is trusted because it resembles an academic citation. A quotation is repeated because the language seems appropriate to the period.

The historian who would never accept an unsigned reminiscence without examining its provenance may accept an AI-generated paragraph without asking where any of its claims originated.

ChatGPT should therefore be treated as an intelligent research assistant, not as a documentary source. It can suggest search terms, identify possible lines of inquiry, compare concepts, organize notes, formulate questions, and help clarify difficult prose. It can also point a researcher toward archives, books, people, and events that merit investigation. Its usefulness, however, does not convert its output into evidence.

The distinction is fundamental. A research assistant may tell a historian that a document probably exists. The historian must still locate and examine it.

Why Fluency Is So Persuasive

Large language models are especially dangerous when they are almost correct. An obviously absurd answer invites skepticism. A plausible answer, containing real names, accurate background information, and one fabricated detail, may pass unnoticed.

ChatGPT’s authority is largely rhetorical. It writes in complete sentences, arranges events chronologically, supplies transitions, and often presents conclusions without visible hesitation. Readers naturally associate these qualities with knowledge. Confidence, specificity, and coherence become substitutes for provenance.

Recent research suggests that this problem is not simply the result of careless prompting. Kalai and his colleagues argue that common accuracy-based evaluations may reward models for guessing instead of acknowledging uncertainty. If a benchmark gives credit only for correct answers and imposes no meaningful cost for plausible errors, the model is encouraged to answer even when abstention would be more responsible. The researchers propose open scoring rubrics that disclose how errors and abstentions will be evaluated.

This observation has important consequences for historians. A model that always produces an answer may appear more useful than one that frequently says, “I do not have enough evidence.” In historical research, however, an admission of uncertainty may be the more accurate and intellectually responsible response.

The historian must therefore evaluate not only what the system says, but whether the system should have answered at all.

Five Forms of Historical Hallucination

Historical hallucinations frequently assume recognizable forms.

The first is the invented citation. ChatGPT may produce a realistic book title, journal article, archival collection, author, volume number, or page range. Every component may appear academically credible even though the source does not exist.

The second is the misquotation of a primary source. A model may modernize, compress, or reconstruct a statement and then present the resulting language inside quotation marks. The general sentiment may be accurate, but the words are not documentary evidence.

The third is the composite biography. Details belonging to several individuals may be joined into one apparently coherent life. This is especially dangerous when researching people who share names, occupations, institutions, military units, or geographic locations.

The fourth is incorrect chronology. A model may identify real events but arrange them in the wrong order, confuse the date of an appointment with the date of an election, or place a person at an institution before that person arrived there.

The fifth is false causation. ChatGPT may connect two events with phrases such as “therefore,” “as a result,” or “this led to,” even when the evidence establishes only sequence or correlation.

These errors are not equally visible. An invented person may be discovered quickly. A subtle chronological error or unsupported causal inference may survive several rounds of editing because it fits an expected narrative.

Why Better Prompts Are Not Enough

Prompt design can improve an AI response. A historian can request citations, ask the system to distinguish facts from interpretations, require expressions of uncertainty, or instruct it not to invent missing information. Such practices are worthwhile.

They are not verification.

A carefully written prompt may reduce the probability of an error, but it cannot transform generated language into historical evidence. Even a response that includes citations must be checked against the cited material. Even a quotation accompanied by a page number must be located on that page. Even a claim presented as certain must be corroborated.

Research on hallucinations in academic writing identifies fabricated citations, factual distortion, named-entity errors, logical inconsistency, and propagation errors as threats to scholarly integrity. The final responsibility remains with the human author. An historian cannot excuse a false statement by explaining that ChatGPT supplied it.

Better prompting is therefore a research technique, not an evidentiary standard.

Testing AI as Historians Test Evidence

Historians need a repeatable method for testing AI-generated research. The evaluation should move beyond impressions such as “the answer seemed good” or “the model performed well.”

A practical scoring rubric might assess eight dimensions:

1.      Citation existence: Does every cited work or archival collection exist?

2.      Citation accuracy: Does the cited source support the specific claim?

3.      Quotation fidelity: Are quoted words reproduced exactly and in context?

4.      Identity accuracy: Have people with similar names or careers been distinguished?

5.      Chronological accuracy: Are dates and sequences correct?

6.      Geographic and institutional accuracy: Are places, offices, organizations, and jurisdictions correctly identified?

7.      Provenance: Can each important assertion be traced to accessible evidence?

8.      Historiographical consistency: Does the answer acknowledge significant scholarly disagreement?

Each category could be scored from zero to four. A zero would indicate fabrication or complete failure. A score of one would reflect major errors. Two would indicate partial support or substantial ambiguity. Three would represent generally accurate work with minor deficiencies. Four would require full and independently verified support.

A weighted rubric would assign greater penalties to invented citations, false quotations, and merged identities than to minor stylistic or contextual omissions. This matters because not all errors have the same scholarly consequence.

Benchmark testing should also use a fixed set of questions rather than memorable anecdotes. Researchers could create a collection of historical questions at several difficulty levels: well-documented national events, obscure local events, disputed interpretations, biographical identity problems, chronological puzzles, and questions whose correct answer is genuinely unknown.

The same questions could then be submitted to several AI systems under controlled conditions. Evaluators would record accuracy, citation validity, unsupported assertions, appropriate abstentions, and changes across repeated runs. Such testing would reveal not only whether a model produces correct answers, but how it fails.

What RAG Can and Cannot Do

Retrieval-augmented generation, commonly known as RAG, offers one method of improving reliability. A RAG system retrieves documents from a designated collection before generating an answer. For historians, that collection might contain archival finding aids, digitized newspapers, oral-history transcripts, government records, or scholarly articles.

This approach can make AI responses more transparent and more closely connected to identifiable sources. It may reduce fabricated citations and permit researchers to inspect the evidence used in producing an answer.

Yet RAG does not eliminate the need for historical judgment.

Recent research distinguishes factuality from faithfulness. A statement may be faithful to a retrieved document while the document itself is inaccurate, biased, incomplete, or misinterpreted. Conversely, a statement may be factually correct but unsupported by the particular documents retrieved. A serious evaluation must test both dimensions.

For historians, retrieval is only the beginning. The fact that a system retrieved a newspaper article does not establish that the article was truthful. The fact that a claim appears in a memoir does not establish that memory was reliable. The fact that several sources repeat a statement does not prove they were independent.

RAG can retrieve evidence. It cannot perform the full intellectual work of source criticism.

The Historian’s Continuing Responsibility

ChatGPT does not threaten historical scholarship simply because it occasionally invents facts. Historical scholarship has always confronted error, forgery, propaganda, partial memory, and misleading testimony.

The greater danger arises when historians allow the fluency of artificial intelligence to replace the discipline’s oldest habit: skepticism.

For centuries, historians have questioned manuscripts, memoirs, newspapers, photographs, statistics, and archives. Artificial intelligence deserves no exemption from that tradition. Its output should be tested, scored, corroborated, and traced to evidence.

The historian’s task remains unchanged. It is not merely to discover information or compose a persuasive narrative. It is to determine what the surviving evidence permits us to say, what remains uncertain, and why the distinction matters.

References

1. Kalai, Adam Tauman, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. “Evaluating Large Language Models for Accuracy Incentivizes Hallucinations.” Nature 653 (2026): 1047–1051. https://www.nature.com/articles/s41586-026-10549-w
2. Rahman, Subhey Sadi, et al. “Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models.” Artificial Intelligence Review 59 (2026), article  70. https://link.springer.com/article/10.1007/s10462-025-11454-w
3. Fadeeva, Ekaterina, et al. “Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval-Augmented Generation.” Findings of the Association for Computational Linguistics: ACL 2026 (2026): 6814–6836. https://aclanthology.org/2026.findings-acl.338
4. Giray, Louie, Md Emdadul Islam, and John Arvin Glo. “AI Hallucinations in Academic Writing: Implications for Research Integrity.” Naunyn-Schmiedeberg’s Archives of Pharmacology (2026). https://pubmed.ncbi.nlm.nih.gov/42217043 

What is RAG ?

What Is Retrieval-Augmented Generation (RAG), and Is It a Groundbreaking Way of Interacting with Historical Documents?

by Alvin Blackshear | Historian & Researcher

For as long as archives have existed, the historian's craft has depended on a single, stubborn skill: finding the right document among thousands of wrong ones. Card catalogs gave way to keyword search engines, which gave way to digitized finding aids, yet the underlying task never changed. A researcher formulates a question, searches a collection using the vocabulary available at the time, and hopes the terms they chose overlap with the terms an archivist or a scanner happened to preserve. Retrieval-Augmented Generation, or RAG, is the first technology in this long lineage that promises to close the gap between what a historian actually wants to know and what a search box is capable of returning. Whether it fully delivers on that promise, and under what conditions, is the question this article sets out to examine.

Why Historians Need RAG Rather Than a Generic Chatbot

It is tempting to assume that any large language model, given enough context, can answer historical questions competently. This assumption is mistaken, and understanding why is essential to understanding what RAG actually contributes.

A generic LLM answers from its training data, a fixed and often outdated snapshot of text scraped from the internet. It has no reliable way to point to a specific archival folder, newspaper issue, or census page as the source of its claim, and it has every incentive, structurally speaking, to produce a fluent and confident answer even when no such source exists in its memory. For a historian, whose entire discipline rests on the traceability of a claim back to a primary document, this is disqualifying. An answer without a citation is not history. It is a plausible-sounding guess.

RAG changes the architecture in a specific way. Instead of relying solely on what a model memorized during training, a RAG system retrieves relevant passages from an external, defined collection, be it a digitized newspaper archive, a set of scanned letters, or a database of military records, and then asks the language model to generate an answer grounded in those retrieved passages. The research group behind the paper "Improving Access to Historical Archives with Real-time RAG-based Systems," led by Stergios Konstantinidis and colleagues, demonstrates this architecture at scale, applying it to a Swiss newspaper archive of roughly half a million segments spanning nearly two and a half centuries. Their system pairs a semantic retrieval and reranking pipeline with grounded answer generation, which means the model's output is tethered to specific archival text rather than floating free of any documentary anchor. This is the essential difference that makes RAG, rather than an ordinary chatbot, the appropriate tool for archival research.

How RAG Differs from Conventional Archival Search

Traditional archival search tools, including many of the platforms libraries and museums have relied on for decades, are built on lexical matching. A researcher types a term, and the system returns documents containing that exact term or a close variant. This works reasonably well when the researcher already knows the vocabulary used by the source material, but historical vocabulary shifts constantly. Place names change. Institutional terminology evolves. Spelling was inconsistent long before anyone standardized it. A search for a term as it is understood today may simply miss a document that describes the same event using nineteenth-century phrasing.

RAG systems address this through semantic retrieval, which represents both the query and the archival text as vectors in a shared conceptual space, allowing the system to surface documents that are meaningfully related even when the wording differs substantially. A master's thesis from the University of Twente, authored by Gauri Bhatnagar and focused on the CLARIAH Media Suite, an audiovisual heritage collection, tested exactly this proposition with real humanities scholars and archival professionals. The study found that RAG genuinely helps with cross-lingual access and exploratory, interpretive queries that lexical search struggles with, but it also surfaced a critical limitation: the system performed well on factual retrieval and named-entity identification, while struggling with interpretive reasoning and the kind of complex synthesis that sits at the heart of humanities scholarship. In other words, RAG can help a historian find the needle, but it cannot yet be trusted to explain what the needle means within a broader historiographical argument.

A separate, more industry-facing piece published by Veridian Software, a company that builds digital newspaper and archive platforms, frames RAG in more practical terms for institutions considering adoption, describing it as a natural extension of the search tools archives already maintain rather than a wholesale replacement of them.

Common Failure Modes in Archival RAG Systems

No technology arrives without its own characteristic failures, and archival RAG is no exception. Several recurring problems deserve explicit attention from anyone applying these tools to historical research.

OCR errors.  Most archival text exists as scanned images that have been converted to machine-readable text through optical character recognition, a process that is notoriously imperfect on older, damaged, or low-quality print. The Konstantinidis paper directly measures this problem and reports that an LLM-based refinement step reduced character error rates by up to roughly forty-five percent and word error rates by close to sixty-one percent, compared to the raw OCR output. That is a substantial improvement, but it is not perfection, and any residual error can silently distort a retrieved passage or an answer built on top of it.

Incomplete retrieval.  Even the best semantic search will not surface every relevant document, particularly when a collection is heterogeneous or when relevant material uses unusual phrasing, an obscure genre, or a minority language. A RAG system that returns five confident-sounding sources may be quietly ignoring a sixth that contradicts them.

Provenance loss.  When a language model synthesizes multiple retrieved passages into a single fluent answer, it is easy for the boundary between one source and another to blur. A historian needs to know precisely which claim came from which document, on which page, from which collection. A system that merges sources into an undifferentiated paragraph has failed a basic archival standard even if every individual fact happens to be accurate.

Temporal confusion.  Archives frequently contain material spanning centuries, and a retrieval system optimized purely for topical similarity has no inherent understanding of chronological sequence. It is entirely possible for a RAG system to present an 1850s account and a 1950s retrospective as though they were contemporaneous, a mistake that would mislead any historian relying on the system's framing rather than checking dates independently.

Fabricated citations.  Perhaps the most consequential failure mode is the language model's tendency to generate a citation that looks correct in form, a plausible archive box number or a newspaper date, but does not correspond to any actual retrieved document. Even well-grounded RAG systems are not fully immune to this, especially when a query has no good answer in the underlying collection and the model is inclined to produce something anyway rather than admit the gap.

The Missing Piece: A Historian's Evaluation Standard

Here is where the existing research reveals a striking gap. Every technical paper reviewed for this article evaluates RAG systems using measures native to information retrieval and natural language processing: retrieval precision, recall, embedding quality, OCR accuracy, system latency, and hallucination rate. A recent survey of RAG evaluation methods by Aoran Gan catalogs this landscape comprehensively, and a companion technical guide from Toloka lays out practical metrics for groundedness and answer faithfulness. These are legitimate and necessary measures. But none of them ask the questions a professional historian actually asks when assessing whether a piece of evidence is trustworthy.

Did the system retrieve the most authoritative version of a source, or merely the most textually similar one? Did it note when retrieved evidence contradicted itself, or did it quietly favor the passage that supported a cleaner narrative? Did it preserve the chain of custody and provenance for each claim? Did it distinguish a primary account from a later secondary interpretation of that account? Did it respect the order of events rather than collapsing decades into an undifferentiated blend? Did it accurately synthesize material drawn from separate collections without conflating them? Did it acknowledge uncertainty where the evidence was genuinely ambiguous, rather than resolving that ambiguity for the sake of a tidy answer? And did it surface evidence that was difficult to find but still relevant, rather than defaulting to whatever ranked highest by similarity score?

This gap between technical evaluation and historiographical evaluation is the foundation for what can be called a Historian's AI Evaluation Framework, or HAIEF. Rather than asking whether a RAG system is efficient, HAIEF asks whether it is trustworthy by the standards the historical profession has used for generations: provenance, contextualization, chronology, corroboration, and transparency about uncertainty. A system might score extremely well on NDCG or answer-correctness metrics, the kind reported by Konstantinidis and colleagues, whose reranking pipeline improved NDCG@10 from roughly sixty-six percent to eighty-seven percent, and still fail a historian's basic sniff test if it cannot show its provenance trail or if it silently smooths over contradictory testimony.

Toward a Historian's Testing Framework

A workable HAIEF would need to translate these five values into repeatable, scorable tests. Provenance could be scored by checking whether every claim in a generated answer links back to a specific, verifiable source with collection, folder, and page-level detail. Contextualization could be scored by asking whether the system situates a document within its original archival and historical setting rather than presenting it as free-floating text. Chronology could be tested with queries that require ordering events correctly across a span of decades or centuries. Corroboration could be tested by deliberately including contradictory sources in a test collection and observing whether the system flags the disagreement or resolves it artificially. Transparency about uncertainty could be measured by presenting queries with genuinely unresolved historical questions and checking whether the system hedges appropriately rather than manufacturing false confidence.

Benchmark Collections and a Comparative Rubric

Testing such a framework requires benchmark collections that reflect the actual diversity of archival material historians work with: personal letters and correspondence, census and vital records, digitized newspapers, military service and pension records, oral history transcripts, and photograph collections with accompanying metadata. Each of these genres carries its own provenance conventions and its own failure risks, and a system that performs well on newspapers may perform poorly on oral histories, where nuance, tone, and interviewer influence complicate straightforward retrieval.

A repeatable scoring rubric applying HAIEF criteria could then be used to compare general-purpose assistants such as ChatGPT, Claude, and Gemini against dedicated archival RAG systems purpose-built for a single institution's holdings. Such a comparison would likely show that general-purpose assistants excel at fluent synthesis but lag on provenance transparency, while dedicated archival systems, closer in spirit to the Konstantinidis pipeline, perform better on traceability but may still struggle with the interpretive reasoning that Bhatnagar's thesis identified as a persistent weakness across the board.

Recommendations for Archives Building AI-Assisted Discovery Tools

Institutions developing these tools should treat historiographical evaluation as a first-class requirement rather than an afterthought bolted on after a system already works technically. This means involving historians and archivists directly in system design, not merely as end-user testers after launch. It means building citation and provenance display into the interface itself, so that a user can see, at a glance, exactly which archival item supports each claim. It means deliberately including contradictory or ambiguous material in test collections so that a system's handling of disagreement can be assessed before deployment, rather than discovered by an unlucky researcher months later. And it means resisting the temptation to optimize purely for fluent, confident-sounding answers, since fluency and historical reliability are not the same thing and can, in the worst cases, actively work against one another.

Conclusion

RAG is not a gimmick, and the evidence gathered here, from a large-scale newspaper archive study to a humanities-focused thesis to a growing body of evaluation literature, makes clear that it represents a genuine advance over keyword search for anyone working with digitized historical material. But calling it groundbreaking requires a qualification. It is groundbreaking as a retrieval technology. It is not yet groundbreaking as a historiographical instrument, because none of the existing research has built evaluation standards around the values historians actually rely on to judge evidence. Closing that gap, through a framework like HAIEF, is not a minor refinement. It is the difference between a tool that finds documents quickly and a tool that a historian can actually trust.

____________________________ 

Sources:

Bhatnagar, Gauri. "Enhancing Multimodal Archival Search and Discovery in the Sound and Vision Archive Using Large Language Models and Retrieval-Augmented Generation." Master's thesis, University of Twente, August 2025. https://purl.utwente.nl/essays/108957.

Gan, Aoran. "Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey." arXiv, April 21, 2025. https://doi.org/10.48550/arXiv.2504.14891.

Jancovic, Marek. "RAG for Historians: A Radically New Way of Interacting with Historical Documents." LinkedIn, March 29, 2026. https://www.linkedin.com/pulse/rag-historians-radically-new-way-interacting-marek-jancovic-ew3xe/.

Konstantinidis, Stergios, Hayman Lotfy, Alexis Erne, Faruk Zahiragic, Min-Yen Kan, and Michalis Vlachos. "Improving Access to Historical Archives with Real-time RAG-based Systems." arXiv, July 3, 2026. https://arxiv.org/html/2607.03440v1.

Toloka. "RAG Evaluation: A Technical Guide to Measuring Retrieval-Augmented Generation." Toloka Blog, August 15, 2025. https://toloka.ai/blog/rag-evaluation-a-technical-guide-to-measuring-retrieval-augmented-generation.

Veridian Software. "What Is RAG and How Could It Support Digital Collection Search?" Veridian Software Knowledge Base, June 29, 2025. https://veridiansoftware.com/knowledge-base/a-new-way-to-search-digital-collections-introducing-rag.

I Asked ChatGPT to Find My Professor Profile. It Failed. Here's Why It Matters.

by Alvin Blackshear  |  Historian & Researcher   <ablackshear@gmail.com>

Why Historians Should Never Trust "I Couldn't Find It."

I asked ChatGPT to find my profile on Rate My Professor. It told me, with no hedging at all, that it could not find a professor profile under that name. That answer was wrong. When I added the name of the college where I taught, the model found my profile immediately, in the same database it had just searched a moment before.

When I asked why its first answer had been incorrect, ChatGPT explained that the initial search had been too narrow and that it had overstated the result. It went further, telling me that it had not been appropriate to state definitively that no profile existed. That admission is more interesting than the original mistake. It names, almost exactly, the failure that researchers of every kind run into constantly: absence of evidence quietly becomes evidence of absence, and a system built to retrieve information ends up asserting a negative it has no real basis for.

This is a small experiment. But small experiments are often where large problems become visible. What follows is an attempt to take this one seriously, not as a complaint about a chatbot, but as a case study in how retrieval systems fail, why those failures are harder to catch than ordinary hallucinations, and what a better evaluation standard would look like.

The Failure Was Not a Hallucination

It is worth being precise about what did not happen here. The model did not invent a fake Rate My Professor page. It did not fabricate reviews, ratings, or a biography. Hallucination, in the sense the AI research community usually means it, involves generating content that has no basis in the retrieved evidence or in reality. That is not what happened in this case.

Instead, the model committed a quieter and, in some ways, more dangerous error. It ran an incomplete retrieval, treated the incomplete result as sufficient, and then converted that insufficient evidence into a categorical claim. Four distinct failures stacked on top of each other:

1. The retrieval step was incomplete, since the search used only a name and not the additional identifying detail (the institution) that would have disambiguated the result.
2. The evidence gathered was insufficient to support any firm conclusion, positive or negative.
3. The model's confidence should have been low, given how thin the search actually was.
4. The final answer failed to reflect that low confidence, presenting a tentative non-result as though it were a settled fact.

Hallucinations tend to draw attention because they are often vivid and easy to catch. A fabricated citation or an invented quote can be checked against a source and shown to be false. This kind of error is harder to catch precisely because it looks like careful, responsible behavior. The model appeared to be reporting a fact rather than making one up. It sounded modest. It just was not modest enough about the right thing.

Why This Matters

Strip the incident down to its underlying logic and it reads like this: I searched. I did not find it. Therefore it does not exist. Stated plainly, that reasoning is invalid, and most people would recognize the problem immediately if it were presented as a syllogism in a logic class. Yet this is exactly the shortcut that retrieval systems take when they are not explicitly designed to resist it.

Anyone who has done serious research recognizes this pattern because it is the default condition of research itself. Archives are incomplete. Metadata is inconsistent. Names appear under variant spellings, married names, initials, or transliterations. Sources are catalogued under headings that make sense to an archivist but not to a later researcher. Databases frequently require an additional piece of context, an institution, a date range, a middle name, before they will surface a record that has been sitting there the entire time.

None of this is exotic. It is the ordinary condition of information work. The question is not whether an AI system will encounter it. The question is whether the system is built to recognize when it has hit this wall, or whether it will simply report the wall as an empty room.

Historians Have Solved This Problem for Centuries

There is a useful comparison to be made with historical methodology, which has spent a very long time developing habits of mind for exactly this situation. A historian who cannot find a record does not usually conclude that the record never existed. Instead, the working assumption is procedural: did I search enough archives, are there alternate spellings of this name, did the person's name change through marriage or naturalization, is this source catalogued under a different heading, is there another repository that might hold a more complete record.

The distinction that matters here is between two very different sentences. "It doesn't exist" is a conclusion about the world. "I haven't found evidence yet" is a conclusion about the search. Good historians are trained, often through painful experience, to default to the second sentence and to treat the first as a claim that requires an extraordinary amount of additional work before it can be responsibly made.

This is not a minor stylistic habit. It is the entire discipline's defense against a kind of error that predates computers by centuries: mistaking the limits of one's own search for the limits of what actually happened.

What Actually Failed

Returning to the four evaluation points named earlier, each deserves a closer look from an evaluator's perspective rather than a user's perspective.

Retrieval was incomplete because the search relied on a single identifier, a name, in a database where names alone are frequently ambiguous or thin. A single search pass is rarely sufficient in any system where entities can share names, use variant spellings, or be indexed under partial records.

Evidence was insufficient because a single unsuccessful query does not constitute a thorough investigation. One null result from one search string tells a system almost nothing about whether a record exists elsewhere in the same database under a slightly different query.

Confidence should have been low because the retrieval process itself provided no strong signal either way. Low retrieval coverage should map to low confidence in any well calibrated system, and a wide gap between actual coverage and stated confidence is itself a measurable defect.

The final answer should have expressed that uncertainty rather than erasing it. The correct output was never "no profile exists." The correct output was closer to "I could not locate one with the search I ran, and here is what might help me look further."

How a Better AI Should Respond

It is useful to imagine the same interaction handled well. Instead of stating that there is no professor profile, a more careful system might have said something like this: I could not locate one using your name alone. If you can tell me the institution where you taught, I can search more precisely.

Notice what that sentence does. It preserves uncertainty instead of erasing it. It invites the user to supply the missing retrieval key rather than closing the inquiry. It treats the null result as information about the search process, not information about the world. This is a small rewording, but it represents a genuinely different epistemic posture, one that treats retrieval as provisional until proven otherwise.

Why This Matters for RAG Systems

This incident connects directly to a broader question about how Retrieval-Augmented Generation (RAG) systems are built and evaluated. RAG systems are only as good as the retrieval pipeline underneath them, and that pipeline depends on several interacting components: retrieval quality, metadata structure, search strategy, query expansion, ranking, and the underlying coverage of the source material itself.

Recent research on RAG evaluation has tried to formalize exactly this dependency. A comprehensive review of RAG evaluation dimensions by Knollmeyer and colleagues works through how retrieval quality, question types, and evaluation metrics interact, and it makes a case that evaluation frameworks need to look separately at retrieval performance and generation performance rather than treating a RAG system's output as a single undifferentiated block. That separation matters here, because the language model itself did not fail at composition or fluency. The retrieval layer beneath it simply stopped searching too early, and the generation layer then dressed that shortfall up as a confident answer.

Work on uncertainty-aware retrieval reinforces the same point from a different angle. Research on uncertainty-aware dynamic retrieval argues that retrieval decisions should be driven by the model's own uncertainty rather than by fixed, one-shot retrieval rules, so that a system recognizes when its confidence is too low to stop searching. Adaptive multi-source retrieval frameworks make a related argument, showing that incorporating query complexity and confidence-aware fusion allows a system to keep pulling in additional sources precisely in the situations where a single retrieval key, like a name without an institution, is not enough to settle the question. Hypothesis-and-verification approaches to retrieval push this further still, treating an initial null result not as a final answer but as a hypothesis to be tested against additional evidence before any conclusion is reported. Self-evaluation approaches to agentic retrieval add another layer, having the system review its own retrieval strategy before committing to a final answer, which is exactly the internal check that would have caught this failure before it reached the user.

None of this is exotic academic theorizing removed from practical stakes. It describes, almost mechanically, what would have needed to happen for ChatGPT to say something more honest than "no profile exists."

An AI Evaluation Rubric

It is possible to turn this single incident into a reusable evaluation checklist, one that applies whenever an AI system reports that something cannot be found. Work on faithfulness-aware uncertainty quantification for fact-checking RAG outputs provides useful scaffolding here, since it separates the question of whether an answer is faithful to its retrieved evidence from the separate question of whether that evidence is factually sufficient in the first place. Applied to a "not found" claim, the following questions become a practical audit:

Were multiple search strategies attempted, or did the system stop after a single query. Were alternate identifiers used, such as an institution, a date, or a variant spelling. Was metadata expanded to account for inconsistent cataloguing. Was uncertainty reported explicitly, rather than folded silently into a flat statement. Did the model distinguish between "not found" and "does not exist." Could one additional retrieval key plausibly change the result.

That last question turned out to be the entire story in this case. One additional identifier, the name of the institution, was all it took to convert a false negative into a correct answer. A system that had asked itself that question before answering would have caught its own mistake.

Implications Beyond Historians

This failure mode is not confined to historians looking up their own teaching profiles. It shows up wherever retrieval sits underneath a serious decision. Legal researchers checking whether a precedent exists face the same risk of concluding "no such case" from an incomplete search of case law. Genealogists working with inconsistent historical spellings run into it constantly. Medical literature searches can miss relevant studies indexed under different terminology, with consequences far more serious than a mistaken professor lookup. Journalists checking whether a claim has prior reporting, researchers conducting literature reviews, and enterprise knowledge bases used inside companies all depend on retrieval systems that can quietly convert "I didn't find it in my index" into "it isn't true." A recent Nature article on synthesizing scientific literature with retrieval-augmented language models makes a related point in the scholarly context specifically, arguing that trustworthy synthesis of academic evidence depends on retrieval systems that are transparent about their own coverage limits rather than presenting synthesized answers as though they rested on complete evidence.

In every one of these domains, the underlying mechanism is identical to what happened with a single Rate My Professor search. One missing retrieval key can completely change the answer, and a system that does not know how to say "I haven't looked hard enough yet" will eventually say something false with total confidence.

Conclusion

Return to where this started. ChatGPT eventually found my profile once I supplied the one piece of context it needed, the institution where I taught. The information had been there the entire time. The system's retrieval capability was never really in question. What failed was something quieter and, in the end, more consequential: the model treated an incomplete search as a complete answer and reported a negative conclusion before it had exhausted reasonable ways of looking.

The most important lesson here was not that ChatGPT failed to find my professor profile. It was that, for a moment, it mistook an incomplete search for a finished one. For historians, for researchers in any field, and for anyone building AI systems meant to be trusted with real questions, that distinction is not a technicality. It is the whole matter.

---

References

Asai, Akari, et al. "Synthesizing Scientific Literature with Retrieval-Augmented Language Models." Nature (2026). https://www.nature.com/articles/s41586-025-10072-4

Fadeeva, Ekaterina, et al. "Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval-Augmented Generation." Findings of ACL 2026. https://aclanthology.org/2026.findings-acl.338

Knollmeyer, Simon, et al. "Evaluating Retrieval Augmented Generation: A Comprehensive Review of Evaluation Dimensions, Question Types, and Application." SN Computer Science 7 (2026). https://link.springer.com/article/10.1007/s42979-026-05134-x

Wu, Haiyan, et al. "Reflective RAG: Self-Evaluation Driven Strategy Optimization in Agentic Retrieval-Augmented Generation." Findings of ACL 2026. https://aclanthology.org/2026.findings-acl.648

"AdaKG-RAG: Adaptive KG-guided Retrieval-Augmented Generation with Hypothesis-and-Verification Retrieval for Multi-hop Question Answering." Journal of King Saud University Computer and Information Sciences (2026). https://link.springer.com/article/10.1007/s44443-026-01028-3

"UA-RAG: Uncertainty-aware Dynamic Retrieval-Augmented Generation." Neurocomputing (2026). https://www.sciencedirect.com/science/article/pii/S0925231226017558