Evaluating Historical Accuracy in LLM: Reconstruction, Jim Crow, and the Civil Rights Movement as High-Stakes Test Cases

by Alvin Blackshear  |  Historian & Researcher   <ablackshear@gmail.com>

Ask a large language model a seemingly straightforward question: "Who was the first African American elected to statewide office after Reconstruction?" The answer arrives instantly, fluently, and with the trappings of scholarship. It may cite books. It may offer dates with confident precision. It sounds, in short, like the work of someone who knows the field.

Historians know better. A question that appears simple on its surface is, in practice, layered with complications. What counts as "statewide office"? Does an elected superintendent of education count the same as a lieutenant governor? Does the answer depend on which state, since Reconstruction unfolded unevenly across the former Confederacy and border states? Does "first" mean first elected, first to take office, or first to serve a full term amid the violence and fraud that often interrupted Reconstruction-era elections? A model that produces a single, tidy name has already made a series of interpretive choices, usually without disclosing them, and sometimes without any grounding in verifiable sources at all.

This is the central argument of this article: historical accuracy cannot be measured merely by whether an isolated fact happens to be correct. Accuracy in historical writing is a compound of chronology, sourcing, context, and interpretation, and any evaluation of AI-generated history that stops at fact-checking a name or a date has already missed most of what makes history rigorous. Reconstruction, African American history, and the Civil Rights Movement make an unusually demanding proving ground for this kind of evaluation, and examining them closely offers historians a template for judging AI output more broadly.

Why These Three Fields Are an Ideal Stress Test for LLMs

Several features of these fields combine to expose the weaknesses of language models more sharply than most other historical subjects.

Scholarship on Reconstruction and the Civil Rights Movement has changed rapidly over the past two generations, as historians have moved from older, dismissive interpretations toward accounts that take Black political agency, local organizing, and grassroots activism seriously. A model trained on a broad mixture of older and newer sources may blend outdated framings with current scholarship without signaling the shift, producing a synthesis that never actually existed in the historiography.

These fields are also politically contested in ways that shape how information about them circulates online, in textbooks, and in public commentary. Terminology is inconsistent across time and place: the language used for racial categories, political offices, and organizations has shifted across the nineteenth and twentieth centuries, and even among historians writing in the same decade. Local, state, and federal histories overlap and sometimes contradict one another, particularly during Reconstruction, when a county-level fusion government might operate differently from the state legislature above it. Archival records are incomplete, and the voices of the enslaved, the formerly enslaved, and the disenfranchised are systematically underrepresented in the documentary record that survives.

Together, these characteristics force a model to reason about evidence rather than simply retrieve settled facts. That is precisely why they make such a useful stress test. A model that performs well on a straightforward question about, say, the dates of a well-documented battle may perform far worse when the underlying historical reality is genuinely contested, sparsely documented, or subject to ongoing revision.

What Does "Historical Accuracy" Actually Mean?

Before any evaluation can proceed, it helps to unbundle the concept of accuracy into its component parts.

Factual accuracy is the most familiar category and covers names, dates, and places. It is necessary but far from sufficient. Chronological accuracy concerns the sequence of events, their timing relative to one another, and the crucial distinction between cause and consequence, a distinction that language models frequently blur when they narrate history as a smooth chain of inevitabilities rather than a contested and contingent process.

Citation accuracy asks two separate questions: do the cited works actually exist, and, if they do, do they actually support the claim attributed to them? A hallucinated citation is an obvious failure, but a real citation attached to a claim it does not support is a subtler and in some ways more dangerous one, since it can survive a cursory check.

Contextual accuracy asks whether the historical setting has been correctly explained, not merely gestured at. Historiographical accuracy asks whether the answer acknowledges that historians disagree, and whether it fairly represents the range of interpretation rather than presenting one school of thought as settled consensus. Evidentiary accuracy, finally, asks whether every important claim can be traced back to documentary evidence, the standard that separates history from well-informed narrative.

A model can satisfy the first of these categories while failing all the others. That gap is where most of the risk in AI-generated history lives.

Common Failure Modes in African American History

Rather than cataloguing hallucinations as isolated errors, it is more useful to organize failures by the type of historical reasoning they violate.

Invented primary sources and fabricated quotations are the most conspicuous failures, and they are especially damaging in African American history, where authentic primary source material is already scarce and precious. A model that invents a plausible-sounding letter or speech does more than get a fact wrong; it pollutes a record that is already thin.

Composite biographies are a subtler failure, in which a model merges the experiences or achievements of two or more historical figures into a single, internally inconsistent account. Chronological compression, in which distinct decades are folded together as though they were a single moment, is common in narratives that span the long civil rights era, a period historians increasingly define as stretching from the 1930s through the 1970s rather than the traditional decade between 1955 and 1965.

Misidentified officeholders and geographic confusion, particularly the conflation of state and local officials, recur often enough in Reconstruction-era queries to be treated as a distinct category. False causal claims, in which one event is assumed to have directly produced another without the intervening political and social process that actually connected them, distort the texture of change over time. Presentism, the projection of modern assumptions and categories backward onto historical actors who did not share them, is perhaps the most pervasive failure of all, since it can infect an otherwise factually accurate answer with an interpretive frame the historical subjects themselves would not have recognized.

Reconstruction as an AI Evaluation Benchmark

Reconstruction is an unusually demanding benchmark because it compresses so much complexity into roughly a dozen years. A model addressing this period correctly must integrate multiple constitutional amendments, uneven state-by-state implementation, rapid and often reversed political change, contested definitions of citizenship and suffrage, shifting racial classifications, waves of violent resistance, and a series of short-lived governments that rose and fell within a single decade.

The documentary base compounds the difficulty. Freedmen's Bureau records, election returns, and the proceedings of state constitutional conventions are voluminous but unevenly digitized, unevenly indexed, and often contradictory on points of basic fact, such as the precise vote count in a contested election. Economic historians examining Reconstruction have shown just how tangled the causal picture becomes when one moves past the political narrative into the socio-economic effects of emancipation and federal enforcement, where multiple datasets, regional variation, and the halting reach of federal authority all interact in ways that resist simple summary. A model asked to explain "the economic effects of Reconstruction" without acknowledging this complexity is not simplifying for clarity; it is producing a distortion.

The Civil Rights Movement as a Verification Challenge

The Civil Rights Movement presents a different but related problem: the temptation toward a tidy, celebratory narrative organized around a small number of famous leaders. Language models trained on the volume of public-facing material about the movement tend to reduce decades of organizing to a handful of familiar names and set-piece events, at the expense of the local activism, particularly the labor organizing, legal work, and grassroots mobilization carried out largely by women and by people whose names never entered the national press.

This tendency shows up as compressed decades folded into a single narrative arc, confusion between overlapping organizations with similar acronyms and overlapping memberships, legal decisions misplaced in time or attributed to the wrong court, invented speeches, and misquoted newspapers. What makes this category of error especially important for historians to flag is that omission functions here as a form of distortion. A summary that mentions only the most famous figures and set-piece protests is not merely incomplete; it actively misrepresents how change actually happened, and it can be just as misleading as an outright fabrication, even though it contains no single false statement.

A Historian's Evaluation Rubric for LLM Output

Translating these concerns into a working tool for evaluation yields a rubric that can be applied consistently across outputs:

Criterion	Questions
Citation authenticity	Does every cited work exist?
Primary-source fidelity	Are quotations verifiable?
Chronology	Is the timeline internally consistent?
Context	Is sufficient historical context provided?
Historiography	Are competing interpretations acknowledged?
Provenance	Can factual claims be traced?
Geographic precision	Are jurisdictions correctly distinguished?
Uncertainty	Does the model acknowledge ambiguity?
Reproducibility	Can another historian repeat the verification?

Applied consistently, this rubric shifts the evaluator's attention away from spot-checking isolated facts and toward the structural qualities that make a historical account trustworthy. A response can score reasonably well on factual accuracy and still fail on historiography, provenance, or uncertainty, and it is precisely those failures that a fact-focused benchmark would miss.

Measuring Historical Accuracy Beyond Benchmarks

Most existing AI benchmarks evaluate factual recall, question answering, and multiple-choice accuracy, formats borrowed largely from standardized testing rather than from historical practice. Recent scholarly efforts have begun to push against this default. One 2026 benchmark built around the Chinese imperial examination tradition was designed specifically to test historical reasoning rather than simple retrieval, and its authors found that even the most capable models struggled once the task moved beyond fact lookup into the kind of analytical work professional historians actually perform. That finding is instructive: models can appear highly competent on narrow factual questions while still falling well short of the reasoning historians rely on day to day.

Historians evaluate evidentiary reasoning, provenance, interpretation, source criticism, contextualization, and the state of competing scholarship. These are fundamentally different evaluation targets than the ones most AI benchmarks were built to measure, and the gap between the two helps explain why a model can pass conventional accuracy tests while still producing what one recent essay memorably termed "stochastic history," an account that carries the surface texture of scholarship without the interpretive reasoning that actually produces it. Closing that gap will require benchmarks designed by historians, for historical reasoning specifically, rather than benchmarks adapted from unrelated domains.

Human-in-the-Loop Historical Verification

None of this argues for abandoning AI as a research aid. It argues for a disciplined workflow in which the historian remains the final authority. A workable sequence looks like this: the model generates a draft; the historian validates every citation; primary sources are independently verified; chronology is checked against the documentary record; competing interpretations are compared against current historiography; the historiographical framing itself is reviewed for balance; and only then does the material move toward publication.

In this workflow, the historian is not a passive consumer of AI output but an active evaluator standing between a plausible draft and a trustworthy account. That distinction, subtle as it sounds, is the difference between treating AI output as evidence requiring evaluation and treating it, mistakenly, as evidence in its own right.

Future Directions

Several developments now underway may narrow the gap between AI fluency and historical rigor. Retrieval-augmented generation systems, which ground model output in retrieved documents rather than internalized patterns, offer one promising path, as do archival retrieval systems built specifically for historical collections. Provenance-aware AI, designed to track and disclose the origin of every claim it makes, addresses the citation problem directly rather than leaving it to after-the-fact verification. Uncertainty estimation, which would allow a model to signal when the historical record is genuinely contested rather than presenting contested claims with false confidence, speaks directly to one of the recurring failures described above. Citation-grounded generation, historian-designed benchmarks, and dedicated historical evaluation datasets round out a research agenda that treats historical reasoning as its own discipline rather than a subset of general knowledge retrieval.

None of these tools eliminate the historian's role. If anything, they raise the bar for what that role requires: historians will need to evaluate not only the answers a retrieval-augmented system produces but the quality, provenance, and interpretive framing of the evidence it retrieves in the first place. A flawed archive fed into a well-engineered retrieval system still produces a flawed history.

Until AI systems consistently meet the standard historians already hold themselves to, historical accuracy will remain not a property inherent to the model, but the product of rigorous, sustained human evaluation.

Sources

"Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination." ACL 2026. https://aclanthology.org/2026.acl-long.1378

"Using Generative AI in Historical Practice." Cambridge University Press, 2026. https://www.cambridge.org/core/elements/using-generative-ai-in-historical-practice/7C1392A6E9DBD379FAA42E2D16A6D45B

"Writing Against the Machine: Computational Authorship and Historical Writing." History (Wiley), 2026. https://onlinelibrary.wiley.com/doi/10.1111/1468-229x.70092

"The Algorithm of Silence: Artificial Intelligence, Archival Bias and the Ethical Reconstruction of Digital Memory." Journal of Documentation, 2026. https://doi.org/10.1108/JD-08-2025-0249

"Automating the Past: Artificial Intelligence and the Next Frontiers of Digital History." International Journal of Humanities and Arts Computing, 2026. https://doi.org/10.3366/ijhac.2026.0361

"Reframing Historical Text Extraction: A Cross-Pathway Validation of OCR, LLM-Assisted Correction, and Direct Multimodal Transcription." Information (MDPI), 2026. https://www.mdpi.com/2078-2489/17/8/722

"Was Freedom Road a Dead End? Socio-economic Effects of Reconstruction in the American South." Economic History Review, 2026. https://onlinelibrary.wiley.com/doi/10.1111/ehr.70085

"Preserving Historical Truth: Detecting Historical Revisionism in Large Language Models." 2026. https://arxiv.org/abs/2602.17433

AI Fact Verification for Historians: Building a Discipline-Specific Protocol for a New Research Environment

by Alvin Blackshear | Historian & Researcher  <ablackshear@gmail.com>

Historians have always worked at the boundary between evidence and interpretation, testing claims against sources, weighing provenance, and living comfortably with uncertainty. Generative artificial intelligence has entered this terrain quickly and unevenly, offering historians an extraordinary tool for discovery while introducing new categories of risk that the discipline has not yet fully reckoned with. The question facing the field is not whether AI belongs in historical research. It already does. The real question is how historians verify what AI produces, and whether existing habits of source criticism are sufficient for a technology that can fabricate footnotes, invent quotations, and drift toward revisionist narratives when prompted to do so.

This article argues that AI can meaningfully accelerate historical research, but it cannot replace professional source criticism, and it proposes the outline of a repeatable verification protocol suited to the particular demands of archival research, primary source authentication, citation validation, chronology checks, and historiographical consistency.

The stakes of getting this right extend beyond any single article or dissertation. History as a discipline depends on a shared confidence that its claims can be traced back to something real, a document, an object, a testimony that survived. When that chain of evidence is compromised, even quietly and even with good intentions, the damage does not stay contained to one footnote. It erodes the credibility of the discipline as a whole, at a moment when public trust in expertise is already under strain. Historians who adopt AI tools without adopting equally rigorous verification habits are not simply risking their own reputations. They are testing the durability of a field that has spent centuries building methods for distinguishing memory from evidence and evidence from invention.

The Promise and the Ceiling

Large language models can summarize secondary literature, suggest starting bibliographies, identify patterns across large text corpora, and help historians locate materials they might otherwise have missed. Used well, these tools compress hours of preliminary searching into minutes. The American Historical Association's Ad Hoc Committee on Artificial Intelligence in History Education, in its 2025 guiding principles, acknowledges this potential while insisting on a firm boundary: generative AI can mimic some of the work historians do, but this should never be mistaken for the work itself. The committee's language is worth sitting with. AI produces texts, images, audio, and video, not truths. It selects words and patterns from training data rather than comprehending the past in its full complexity and contradiction. When a model encounters a gap in its training, it does not pause to acknowledge uncertainty. It fills the gap with something plausible sounding, a process the field now calls hallucination.

For historians, hallucination is not an abstract technical curiosity. It is a direct threat to the evidentiary basis of scholarship. A fabricated citation that looks correctly formatted, attached to a real-sounding archive or journal, can pass a cursory glance and enter a footnote before anyone notices the source does not exist. The AHA's guiding principles are unambiguous on this point: any reference generated by AI must be checked against the original before it is used, and a reference included in a footnote without that check is never acceptable practice, regardless of how the historian frames their working process.

Why Automated Fact-Checking Falls Short

It is tempting to assume that the solution to AI hallucination is simply more AI, in the form of automated fact-checking systems layered on top of generative tools. Recent research complicates that assumption considerably. A 2026 study led by Kelly Amaddio examined how people respond to automated fact-checkers correcting misinformation, using a sample of more than thirteen hundred participants exposed to a piece of gun control misinformation and a subsequent AI-generated correction. The findings revealed a persistent and troubling pattern: prior beliefs shaped both continued endorsement of the misinformation after correction and perceptions of the fact-checking system's reliability. When the fact-checker was described as only moderately accurate, close to the accuracy levels people already associate with existing AI systems, prior beliefs predicted whether the correction actually changed minds. Even when participants were told the system was highly accurate, the effect of prior belief did not disappear, only lessened.

This matters for historians because it demonstrates that automated verification does not operate in a neutral space. It is filtered through the same human biases that shape how any evidence is received, and it can be defeated by claims that already align with what someone wants to believe. Historical scholarship, particularly on contested and politically charged topics, is especially vulnerable to this dynamic. A fact-checking layer bolted onto a language model cannot substitute for the disciplinary training that teaches historians to interrogate their own assumptions as rigorously as they interrogate a source.

Revisionism as a Distinct Problem

Contested historical claims present a challenge that generic fact-checking frameworks were never designed to solve, because the disagreement is not always about a discrete, verifiable fact but about interpretation, framing, and the political stakes attached to memory. A 2026 study by Francesco Ortu and collaborators addresses this directly. The researchers built a benchmark called Historical Misinfo, drawing on five hundred contested historical events from forty five countries, each paired with a documented factual narrative and a documented revisionist counter-narrative. They then tested how large language models responded under neutral prompting compared with prompting that explicitly requested the revisionist version of events.

The results are instructive. Under neutral conditions, most models leaned closer to the factual reference. But when users explicitly requested a revisionist framing, every model tested showed a sharp increase in revisionist output, with little resistance or self-correction. This finding should give historians real pause. It suggests that a model's apparent reliability on a given topic says little about how that same model will behave when a user, deliberately or not, nudges it toward a distorted account. For historians working on genocide, colonialism, contested territorial claims, or any subject where competing national narratives exist, this is not a marginal concern. It is central to responsible use.

What Verification-First Research Looks Like Elsewhere

Historians are not alone in confronting this problem. Related work emerging from adjacent fields offers useful models for how disciplines might restructure their relationship with AI around verification rather than convenience. Lei You, Lele Cao, and Iryna Gurevych have argued that AI-assisted peer review in the sciences should be verification-first rather than review-mimicking, proposing that AI tools function as adversarial auditors generating checkable evidence rather than as systems that simply predict what a human reviewer might conclude. Belinda Mo extends this argument to the broader landscape of autonomous AI research agents, warning of a widening verification gap between what AI systems can produce and what any human can realistically check, and calling for verification infrastructure built around observable workflows and clear attribution.

A study by Raymond Solga and Mohammed Sarwar offers perhaps the closest parallel to the historian's own situation. Comparing traditional historical methods against several generative AI tools in identifying and validating primary sources in ancient history, the researchers found that AI performed well at broad content discovery and thematic synthesis but struggled significantly with genre boundaries, provenance transparency, and the kind of contextual interpretation that trained historians bring almost instinctively to a source. Their proposed solution, a human-in-the-loop framework built around model pluralism and provenance-first protocols, maps closely onto what historians already know from decades of archival practice, simply extended to include AI as one input among many that must be checked rather than trusted.

There is also a temporal dimension worth naming. Carlo Iacono's recent audit of AI-assisted research output found that the models cited in published work are frequently outdated by the time readers encounter them, sometimes by close to a year, with model versions superseded and behaviors shifting in ways that are rarely documented. For historians, this is a reminder that any claim about what a given AI tool can or cannot do should be dated and treated as provisional rather than as a permanent property of the technology.

Toward a Repeatable Protocol

Drawing these threads together, a workable verification protocol for historians should rest on several commitments. Every AI-generated citation must be checked against the primary source or an authoritative secondary source before it appears in any written work, with no exceptions for citations that merely look plausible. Chronology and provenance claims generated by AI require independent confirmation, since these are precisely the areas where current tools show the weakest performance. Contested or politically charged claims deserve heightened scrutiny, given the demonstrated tendency of models to drift toward whatever framing a prompt implies. Verification work should be documented as it happens, noting which tool was used, when, and what was checked, both for the historian's own accountability and because the tools themselves change quickly enough that undocumented claims about their reliability age poorly. And throughout, the historian's own expertise must precede rather than follow AI use, since evaluating a model's output well requires the same depth of subject knowledge that historians have always needed to evaluate any source.

None of this diminishes the real value AI offers historians willing to use it carefully. It does insist that the discipline's oldest habit, treating every claim as something to be verified rather than assumed, remains the right response to a genuinely new technology. Generative AI can accelerate the search for evidence. It cannot substitute for the judgment that turns evidence into history.

There is a version of this moment that history has lived through before, in smaller ways, each time a new technology promised to change how the past is studied. Microfilm, digitization, keyword-searchable databases, all reshaped the pace and reach of archival work without ever touching the core discipline of asking what a source can and cannot tell us. Generative AI is different in degree, not necessarily in kind. It compresses years of searching into minutes, but it also compresses the distance between plausible and true, and that compression is precisely where a historian's training earns its keep. The AHA's committee put it plainly: there are no shortcuts to expertise, and the skills that let a scholar recognize a hallucinated citation or a subtly revisionist framing can only be built through the same sustained engagement with sources that has always defined the craft. A historian who verifies rigorously will find in AI a genuine research partner. A historian who does not will find, sooner or later, that the past they have written no longer holds.

______________________________

Sources

American Historical Association, Ad Hoc Committee on Artificial Intelligence in History Education. "Guiding Principles for Artificial Intelligence in History Education." Approved by AHA Council, July 29, 2025. https://www.historians.org/resource/guiding-principles-for-artificial-intelligence-in-history-education/

Amaddio, Kelly M., et al. "Prior Beliefs and Automated Fact Checking: Limits on the Accuracy of AI Verification." February 5, 2026. https://doi.org/10.1371/journal.pone.0342332

Iacono, Carlo. "The Half-Lives of Generative-AI Evidence." July 27, 2026. https://doi.org/10.48550/arXiv.2607.24032

Mo, Belinda. "The Age of AI Agents Demands a New Scientific Paradigm to Sustain Trustworthy Science." June 18, 2026. https://doi.org/10.48550/arXiv.2607.26064

Ortu, Francesco, et al. "Preserving Historical Truth: Detecting Historical Revisionism in Large Language Models." February 22, 2026. https://doi.org/10.48550/arXiv.2602.17433

Solga, Raymond S., and Mohammed J. Sarwar. "Evaluating Generative AI in Historical Research." AI & Antiquity (2026). https://doi.org/10.64946/aiantiquity.v2i1.003

You, Lei, Lele Cao, and Iryna Gurevych. "Preventing the Collapse of Peer Review Requires Verification-First AI." February 12, 2026. https://doi.org/10.48550/arXiv.2601.16909
 

Beyond Correlation: How Historians Distinguish Causation, Coincidence, and Contingency in the Age of AI

by Alvin Blackshear | Historian & Researcher  <ablackshear@gmail.com>

There is a particular kind of intellectual vertigo that comes from watching a machine narrate the past with total confidence. Ask an AI system to explain why a revolution happened, why a movement gained momentum, or why a community's history unfolded as it did, and it will almost always answer. The prose will be fluent, the timeline will be clean, and the causal claims will arrive stated as fact rather than as argument. What the answer will rarely do is pause to ask whether the events it has linked together were actually connected at all, or whether they simply happened to occur near one another in time. This is the quiet crisis facing historical writing in an era of generative AI: the technology is exceptionally good at producing narratives that sound causal, and considerably less good at knowing whether those narratives are true.

Historians have spent more than a century developing tools to guard against exactly this failure. Long before anyone worried about large language models hallucinating a tidy story out of scattered facts, historians were worried about themselves doing the same thing. The discipline's entire methodological apparatus, footnotes, source criticism, peer review, historiographical debate, exists in large part to slow down the impulse to declare that because one thing followed another, the first caused the second. What AI-generated history reintroduces, at scale and with new fluency, is an old temptation: the temptation to mistake sequence for cause, and pattern for proof.

The Seduction of Sequence

Correlation is easy to notice and hard to interpret responsibly. Two trends move together, two events happen in the same decade, two figures cross paths, and the mind supplies a connection because connection is a more satisfying story than coincidence. Sarah Woodhouse's recent work on correlation reminds us that not all statistical relationships imply causation, and that there are several distinct varieties of correlation, only one of which reflects a genuine causal link between the variables involved.[1] The others might reflect a shared underlying cause, a coincidence of timing, a feedback loop running in the opposite direction from what intuition suggests, or simply noise that resembles a pattern if you squint at it long enough.

Historians face this same interpretive challenge, except their raw material is not a spreadsheet but the messy residue of human lives: letters, court records, newspapers, oral testimony, material culture. When two developments appear together in this record, a rise in labor unrest alongside a shift in immigration policy, a change in religious practice alongside an economic downturn, a legal victory alongside a change in public rhetoric, the historian's task is to ask what actually connects them, if anything does. Johannes Nagel's comparative analysis of causal reasoning in historical scholarship makes a useful observation here: historians rarely operate with the clean methodological categories that philosophers of science might prefer. Real historical practice is often described as vaguely causal-analytical, blending causal-intentional explanation, attention to unintended consequences, and selective use of theory and comparison, without ever fully resolving into a single rigorous method.[2] This is not a failure of the discipline. It is an honest reflection of how difficult the underlying problem actually is.

AI systems, by contrast, do not experience this difficulty as friction. A model trained to produce plausible continuations of text will happily supply a causal mechanism because causal mechanisms make for better sentences than admissions of uncertainty. The danger is not that AI invents facts out of nothing, though it sometimes does that too. The deeper danger is that it takes real facts, arranges them in a narrative shape, and lets the shape itself imply a causal claim that no historian has actually verified.

Mechanism, Not Just Chronology

If there is a single principle that separates serious historical causal argument from mere narrative sequencing, it is the requirement of an explicit mechanism. It is not enough to say that event A preceded event B and therefore contributed to it. A historian is expected to explain how A produced B: through what institutions, what decisions, what material pressures, what beliefs held by which people.

Daniel Little's work on the microfoundations of social explanation is instructive here. Little argues that sweeping historical forces such as capitalism, the state, or industrialization do not themselves act as causal agents in any direct sense. They are abstractions that describe the aggregate result of countless individual decisions, made by people operating within specific institutions and constrained by specific incentives and beliefs. A causal historical explanation, on this view, has to eventually cash out in terms of those individuals: what they knew, what they wanted, what options were actually available to them, and how their choices interacted with those of others to produce a larger outcome. This is demanding work. It requires the historian to move between scales, from the granular texture of a single decision to the broader pattern that many such decisions eventually create, without losing the thread that connects the two.[3]

Mark Hewitson's argument about the marginalization of causal thinking within the historical profession adds an important wrinkle to this picture. Hewitson suggests that the turn toward language, discourse, and representation that reshaped historical writing over the past several decades, whatever its genuine intellectual gains, also made many historians reluctant to speak plainly about why things happened. Causal explanation came to seem naive, or worse, complicit in older positivist ambitions to discover fixed laws of history. And yet, as Hewitson points out, historians never actually stopped making causal arguments. Every account of a revolution, an election, a migration, or a social movement is saturated with implicit claims about why it occurred. What was lost was not the practice of causal reasoning but the willingness to examine that practice openly and hold it to a clear standard.[4]

This matters enormously for evaluating AI-generated history, because a system that has absorbed decades of scholarship in which causal claims are made implicitly, without explicit justification, will likely reproduce that same implicitness. It will offer causal-sounding narratives without ever surfacing the mechanism that supposedly connects cause to effect, because the training material it draws from often does the same thing. The corrective, for both human historians and the tools they now use, is the same: state the mechanism, or admit that you cannot.

What Coincidence and Contingency Demand of Us

Not everything that happens in history happens for a deep structural reason. Some things happen because a particular person was in a particular place at a particular moment, because a letter arrived a day late, because an illness struck one leader and spared another. Historians call this contingency, and distinguishing it from genuine causation is one of the discipline's more delicate tasks.

The stakes of this distinction are not merely academic. When causal narratives are built where only coincidence existed, the resulting history can flatten the genuine unpredictability and agency that shaped real events into something that looks inevitable, as though it could only have happened the way it did. This has particular consequences for histories that have too often been treated as footnotes to a larger, supposedly more central narrative. African American history, feminist history, and the history of gay and queer communities have all, at various points, been subjected to accounts that either erase contingency entirely, presenting outcomes as foregone conclusions of impersonal social forces, or that erase causation entirely, treating hard-won structural change as a matter of isolated coincidence rather than sustained organizing and argument. Getting the distinction right, separating what was truly contingent from what was the product of deliberate, structural, causal effort, is not a technical nicety. It is a matter of according historical actors the accuracy they deserve, whether that means recognizing the genuine unpredictability they navigated or recognizing the genuine causal power of what they built.

Tay Jeong's work on counterfactual reasoning offers historians one of their sharpest tools for making this distinction. The basic move is to ask what would have happened in the absence of the proposed cause. If removing a particular event or decision from the historical record would plausibly have left the outcome largely unchanged, the causal claim weakens considerably. If removing it would plausibly have produced a genuinely different outcome, the causal claim gains support. Jeong's more technical contribution is to note that counterfactual reasoning is most useful not as a blanket method applied indiscriminately, but in specific situations, particularly where two or more candidate causes seem to be doing similar explanatory work and our untrained intuitions about which one truly mattered are likely to mislead us. This is precise, careful reasoning, and it is exactly the kind of reasoning that a fluent AI narrative can obscure simply by never raising the question in the first place.[5]

The Discipline of Alternative Explanations

Good historical argument does not simply assert a cause and move on. It considers what else might explain the same outcome, and it explains why the preferred causal account is more persuasive than the alternatives. This habit of mind, developed across the humanities, has an unexpected ally in fields that seem far removed from history. Hannah Correia and her coauthors, writing about causal inference in environmental research, describe systematic approaches for distinguishing genuine causal relationships from mere observational association, approaches built around identifying confounding variables and establishing plausible mechanisms before accepting a causal claim.[6] Linbo Wang's comparative overview of causal inference frameworks, spanning the traditions associated with Rubin, Pearl, and structural equation modeling, makes a similar point from a more statistical angle: rigorous causal reasoning requires explicit assumptions, and different frameworks make different assumptions visible in different ways.[7] Though these authors are not writing about history, their underlying discipline, of refusing to accept a causal story until alternatives have been seriously weighed, translates directly into the historian's craft.

This is precisely the discipline that AI-generated narratives tend to skip. A well-trained model can produce a fluent account of why a particular reform succeeded, but it rarely volunteers the alternative explanations a careful historian would consider and reject: that the reform succeeded for reasons largely unrelated to the actors typically credited, that its apparent success was partly an artifact of how success was later measured, or that multiple contributing factors were entangled in ways that resist a single tidy story.

Evidence, Uncertainty, and the Historian's Honesty

Underneath all of this sits the question of evidence. Every causal claim a historian makes rests on some body of evidence, and that evidence varies enormously in quality, completeness, and reliability. Xinyue Chen and colleagues, in their study of how historians actually use visualization in their published work, found that historians face persistent practical and epistemological barriers when trying to represent uncertainty and justify their conclusions to readers and peers.[8] The justification burden, as they describe it, is not a peripheral concern. It is central to what makes a piece of historical writing trustworthy.

Georg Iggers, reflecting late in his career on the historian's role as an engaged intellectual, insisted that this burden cannot be set aside even by historians who write from a place of moral or political conviction. Iggers rejected the idea that historians must choose between detached objectivity and partisan advocacy.[9] Instead, he argued for a middle position: historians can and should be engaged with the pressing concerns of their time, but that engagement must never come at the cost of rigorous evidentiary standards. A historian writing feminist history, or African American history, or the history of gay and queer life, is no less bound by these standards than any other scholar, and indeed the stakes of getting the causal story right, of neither overstating structural inevitability nor understating hard-won agency, are often higher precisely because these histories have so frequently been distorted or dismissed.

Frank Ankersmit pushes this even further by reminding us that the past itself no longer exists in any directly accessible form. Historians do not reproduce it; they represent it, using an inherited semantic apparatus of meaning, truth, and reference that Ankersmit argues must be handled with care rather than assumed to work automatically.[10] If historical writing is, at some level, an act of representation rather than simple description, then AI-generated historical narrative inherits this same burden. It too is representing the past, not retrieving it, and representation always involves choices about emphasis, framing, and inclusion that deserve scrutiny rather than passive acceptance.

A Standard Worth Holding

What emerges from this body of scholarship is not a single formula but a disposition, a way of approaching any historical claim, human-authored or machine-generated, with a consistent set of questions. Is there a stated mechanism connecting cause to effect, or only a sequence dressed up as causation? Have alternative explanations been seriously considered and set aside for identifiable reasons? Has the line between structural causation and genuine contingency been drawn carefully, rather than assumed? Would the outcome have looked meaningfully different if the proposed cause were absent? And does the account acknowledge the limits of its own evidence, rather than presenting a confident narrative where the underlying record is thin or contested?

These questions predate artificial intelligence by decades, and they will outlast any particular model or platform. What AI has changed is the volume and fluency with which causal-sounding narratives can now be produced, and therefore the urgency of applying these questions consistently. The historians whose work informs this discussion were not writing with AI in mind. They were writing because the problem of causation in history has always been genuinely hard, and because getting it wrong has always carried real costs, especially for the histories of communities whose stories have too often been told either as inevitable or as accidental, when the truth usually lies in the harder, more interesting space between. That space, between correlation and causation, between coincidence and contingency, between chronology and mechanism, is where careful historical work has always lived. It is also, increasingly, where the responsible use of AI in historical research will need to live too.

---

Notes:

[1] Sarah Woodhouse, "When Two Things Seem Linked (but Aren't): Understanding Different Types of Correlations," Clearer Thinking, July 2, 2025, https://www.clearerthinking.org/post/when-two-things-seem-linked-but-aren-t-understanding-different-types-of-correlations.

[2] Johannes Nagel, "Causal Explanation in History: An Analysis of Research Strategies," Analyse & Kritik, March 12, 2026, https://doi.org/10.1007/s11577-026-01059-8.

[3] Daniel Little, Microfoundations, Method, and Causation: On the Philosophy of the Social Sciences (New Brunswick, NJ: Transaction Publishers, 1998).

[4] Mark Hewitson, History and Causality (Basingstoke: Palgrave Macmillan, 2014).

[5] Tay Jeong, "When to Use Counterfactuals in Causal Historiography," Sage Journals, February 2, 2025, https://doi.org/10.1177/00491241251314039.

[6] Hannah E. Correia et al., "Best Practices for Moving from Correlation to Causation in Environmental Research," Nature Communications, February 24, 2026, https://rdcu.be/fxl0y.

[7] Linbo Wang, "Causal Inference: A Tale of Three Frameworks," Journal of Data Science, February 11, 2026, https://doi.org/10.6339/25-JDS1211.

[8] Xinyue Chen et al., "How Historians Use Visualization: A Corpus-Backed Taxonomy and Analysis for Cross-Disciplinary Practice," July 10, 2026, https://doi.org/10.48550/arXiv.2605.01456.

[9] Georg Iggers, "The Historian as an Engaged Intellectual: Historical Writing and Social Criticism – A Personal Retrospective," in The Engaged Historian: Perspectives on the Intersections of Politics, Activism and the Historical Profession, ed. Stefan Berger (New York: Berghahn Books, 2019), 277–299.

[10] Frank Ankersmit, Historical Representation (Stanford, CA: Stanford University Press, 2002).

---

Works Cited

Ankersmit, Frank. Historical Representation. Stanford, CA: Stanford University Press, 2002.

Chen, Xinyue, et al. "How Historians Use Visualization: A Corpus-Backed Taxonomy and Analysis for Cross-Disciplinary Practice." July 10, 2026. https://doi.org/10.48550/arXiv.2605.01456.

Correia, Hannah E., et al. "Best Practices for Moving from Correlation to Causation in Environmental Research." Nature Communications, February 24, 2026. https://rdcu.be/fxl0y.

Hewitson, Mark. History and Causality. Basingstoke: Palgrave Macmillan, 2014.

Iggers, Georg. "The Historian as an Engaged Intellectual: Historical Writing and Social Criticism – A Personal Retrospective." In The Engaged Historian: Perspectives on the Intersections of Politics, Activism and the Historical Profession, edited by Stefan Berger, 277–299. New York: Berghahn Books, 2019.

International Commission for the History and Theory of Historiography (ICHTH). 2026 Congress Proceedings. https://www.ichth.net/content/home.html.

Jeong, Tay. "When to Use Counterfactuals in Causal Historiography." Sage Journals, February 2, 2025. https://doi.org/10.1177/00491241251314039.

Little, Daniel. Microfoundations, Method, and Causation: On the Philosophy of the Social Sciences. New Brunswick, NJ: Transaction Publishers, 1998.

Nagel, Johannes. "Causal Explanation in History: An Analysis of Research Strategies." Analyse & Kritik, March 12, 2026. https://doi.org/10.1007/s11577-026-01059-8.

Wang, Linbo. "Causal Inference: A Tale of Three Frameworks." Journal of Data Science, February 11, 2026. https://doi.org/10.6339/25-JDS1211.

Woodhouse, Sarah. "When Two Things Seem Linked (but Aren't): Understanding Different Types of Correlations." Clearer Thinking, July 2, 2025. https://www.clearerthinking.org/post/when-two-things-seem-linked-but-aren-t-understanding-different-types-of-correlations.

Ten Common Hallucinations in Genealogical AI

Ten Common Hallucinations in Genealogical AI

by Alvin Blackshear | Historian & Researcher <ablackshear@gmail.com>

If you've started using AI tools to help build out your family tree, you've probably noticed something exciting: AI can search patterns, suggest connections, and summarize records faster than any human researcher. But you may have also noticed something unsettling. Sometimes the AI is simply wrong, and it's wrong with total confidence. It doesn't hedge. It doesn't say "I'm not sure." It states fabricated facts as if they were pulled straight from a courthouse archive.

This isn't a quirk specific to genealogy tools. It's a well-documented feature of how large language models work. Researchers have found that AI systems are, in a sense, rewarded for guessing rather than admitting uncertainty, much like a student on a hard exam who writes down a plausible-sounding answer rather than leaving the question blank. The same dynamic that produces fabricated legal citations in court filings and nonexistent references in academic papers also produces invented ancestors, false marriages, and imaginary historical explanations in your family tree.

For beginning genealogists who are comfortable with AI tools but new to the discipline of genealogical research, this creates a real risk. Family history has its own standards of evidence (rules developed over generations of professional practice), and AI, for all its fluency, doesn't inherently know or follow them. Below are ten (plus a bonus eleventh) hallucinations you're likely to encounter, along with how to spot each one.

#1. Invented Ancestors

The most obvious and probably most common hallucination is the flat-out invented person. If your records show John Smith born in 1846, married to Sarah Jones, and died in 1912, an AI tool may cheerfully add "William Smith, father of John" to your tree, simply because a father-shaped gap exists and the model has learned that fathers usually appear in these slots.

No birth record names him. No probate record mentions him. No document anywhere establishes his existence. He is, in the most literal sense, a statistical guess dressed up as a fact.

How to catch it: Ask the AI directly, "What specific document supports this person's existence?" If it can't name one, or names one that turns out not to exist (see #5), strike the individual from your tree.

#2. False Parentage

This is a subtler and more dangerous error because it doesn't invent anyone; it just connects the right person to the wrong family. Imagine two men named James Brown, both born in Virginia in 1818, but belonging to two entirely unrelated families. An AI tool, pattern-matching on name, place, and approximate date, may merge them, attaching one man's documented parents to the other man's documented children.

Because online family trees copy from each other constantly, an error like this doesn't stay contained. It can spread across dozens or hundreds of public trees within months, becoming "true" simply through repetition.

How to catch it: Whenever two individuals share a name, county, and rough birth year, ask the AI to list the specific evidence distinguishing them: land records, church parishes, associates named as witnesses, distinguishing middle names. If it can't distinguish them, you probably have two people, not one.

#3. Collapsing Two People into One

Related to false parentage, this hallucination happens when the AI treats "same name, similar era, same county" as sufficient proof of "same person." Common names are especially vulnerable: John Williams, William Johnson, Mary Jones, Elizabeth Brown. In many rural communities of the eighteenth and nineteenth centuries, several unrelated people carried the same name at the same time.

How to catch it: Before accepting that two records refer to one individual, ask what independent evidence (land transactions, church membership, military service, or family testimony) ties them together, not just shared name and geography.

#4. Splitting One Person into Several

The mirror-image error: one real person recorded inconsistently, as "Robert," "Robt.," "Bob," and "R.J.," gets treated by the AI as four different people. Suddenly your family tree has artificially multiplied, with duplicate siblings or duplicate spouses cluttering the record.

Nineteenth-century clerks were not consistent spellers, and names were frequently abbreviated, translated, or anglicized. AI models, trained to treat text strings literally, can miss the obvious inference that a human researcher would make instantly.

How to catch it: When a "new" individual appears with a name variant of someone already in your tree, in the same location and timeframe, check whether the records could plausibly describe one person before treating them as separate.

#5. Invented Sources

Perhaps the easiest hallucination to catch once you know to look for it, and one of the most dangerous when you don't. AI tools will sometimes cite a source that sounds entirely plausible: "County Probate Book 14, page 327" or "Virginia Marriage Register, Volume 8." The citation has the right shape, the right format, the right level of specificity. It just doesn't exist.

This mirrors a broader, well-documented problem. A 2026 analysis of AI-generated academic writing found a sharp rise in nonexistent references following widespread adoption of large language models, with a conservative estimate of nearly 147,000 hallucinated citations appearing in 2025 alone. Similar patterns have shown up in legal filings, where courts have caught lawyers submitting briefs built on cases that were never decided because an AI tool invented them. Genealogy is not immune to the same failure mode; if anything, it may be more vulnerable, since archival citation formats are formulaic and easy for a model to imitate convincingly.

How to catch it: Never accept a citation at face value. Look the source up yourself, in the actual archive, library catalog, or online record collection, before recording it as fact. If you can't locate it, assume it doesn't exist until proven otherwise.

#6. Imaginary Historical Context

AI loves to explain. "The family likely moved west because of the Panic of 1837." It's a tidy, plausible sentence. It may even be historically reasonable. But plausible is not the same as documented, and an AI tool rarely marks the difference clearly. Historical interpretation, offered without evidence specific to your family, quietly becomes historical fiction wearing the authority of fact.

How to catch it: Treat any sentence containing words like "likely," "probably," or "given the context of the time" as a hypothesis to test, not a fact to record. Ask what document (a land sale, a migration record, a letter) actually supports the claimed motive.

#7. Overconfident Translation

If your ancestors left records in German, Polish, Latin, French, or Dutch, AI translation can feel like a superpower, until it isn't. AI models translate old church and civil records with impressive fluency, but that fluency can mask serious errors: invented words to fill illegible gaps, misread names, or entire phrases reconstructed from context rather than actually read. Genealogists working with foreign-language parish records have reported AI tools fabricating names, places, and even whole narrative details that don't exist in the original document.

How to catch it: For any translation affecting a name, date, or relationship, ask for a word-by-word rendering alongside the smooth translation, and compare it against the original image yourself, or have a human speaker of the language check ambiguous passages.

#8. Fabricated Relationships

A record lists a witness, Samuel Jones, at a wedding or a will signing. The AI concludes he must be an uncle, brother, father, or cousin. But witnesses in historical documents were frequently neighbors, business associates, fellow church members, or simply friends, not relatives at all. The AI's eagerness to complete the family picture leads it to assign a relationship where the record specifies none.

How to catch it: Treat every unlabeled name in a record (witness, executor, sponsor, bondsman) as a relationship of unknown type until independent evidence establishes otherwise.

#9. Timeline Repair

Large language models dislike contradictions, and historical records are full of them. If a record shows a birth in 1820 and a marriage in 1832, and something else nearby seems inconsistent, an AI tool may quietly adjust one of the dates so everything appears internally consistent. The apparent contradiction vanishes, but so does the actual historical evidence, replaced by a smoothed-over fiction that fits the model's sense of what "should" be true.

How to catch it: When a date changes between one AI response and another, ask why, and check both figures against the original record. Real historical records are often genuinely inconsistent, and that inconsistency is itself information, not a bug to be silently fixed.

#10. Source Laundering

This is probably the least recognized hallucination, and arguably the most insidious. AI tools trained on the vast ocean of user-submitted online family trees absorb whatever errors already exist there, including all the mistakes described above, made by human researchers over the past two decades. The AI then presents those errors back to you as though they were settled historical fact.

Nothing here is invented from scratch. That's what makes it dangerous. The information already circulates online, attached to seemingly credible trees and family history websites, so it carries an unearned appearance of authority. The AI hasn't lied exactly; it has simply repeated community folklore as though it were consensus.

How to catch it: Be especially skeptical of any claim the AI attributes to "commonly accepted" genealogy or "widely documented" family history. Ask what primary source underlies the claim, not what other online trees say about it.

An Eleventh Hallucination: Certainty Inflation

There's one more pattern worth naming, even though it's rarely discussed: certainty inflation. Professional genealogists are trained to say, when the evidence doesn't fully support a conclusion, "the evidence is insufficient." AI tools are much less comfortable with that kind of humility. Instead, they tend to write things like "based on available evidence, John was almost certainly the son of..."

That phrase, "almost certainly," can represent an enormous leap well beyond what the underlying evidence supports. This isn't accidental. Research into why language models hallucinate has found that the training and evaluation processes used to build these systems reward confident, plausible-sounding answers over honest expressions of uncertainty, the AI equivalent of a student guessing on a test rather than leaving a blank. The result is a model that is statistically inclined to sound more certain than the facts warrant.

Toward a Genealogical AI Evaluation Framework

None of this means AI is useless for family history research; far from it. It means AI output needs to be treated the way a careful historian treats any single source: as a claim to be verified, not a fact to be recorded.

One useful discipline is to treat every AI-generated genealogical statement as a research hypothesis, never as evidence in itself. Accept only what can be traced back to an identifiable primary or reliable secondary source, and preserve uncertainty wherever the historical record itself remains uncertain. In practice, that means running each AI claim through a short mental checklist:

Criterion	Key Question
Source authenticity	Is every factual claim traceable to a real, locatable source?
Evidence chain	Can the conclusion be reconstructed from the cited evidence?
Identity confidence	Has the AI distinguished between similarly named individuals?
Chronological consistency	Are dates plausible without silent correction?
Geographic plausibility	Do locations fit historical boundaries and migration patterns?
Uncertainty disclosure	Does the AI distinguish evidence from inference?
Genealogical Proof Standard alignment	Would the conclusion satisfy professional genealogical standards?

Applying a framework like this won't eliminate hallucinations (no current AI system is free of them), but it will keep you from mistaking a fluent, confident answer for a well-documented one. AI can be a genuinely powerful research assistant for genealogy: it can suggest search strategies, summarize long documents, translate unfamiliar scripts, and spot patterns across large record sets faster than any human could alone. Just remember that its job is to help you find the evidence, not to become the evidence itself.

________________________________

Sources:

Kalai, Adam, et al. "Why Language Models Hallucinate." arXiv, 4 Sept. 2025, https://arxiv.org/abs/2509.04664.

Newsham, Jack. "AI Hallucinations in Court Documents Are a Growing Problem, and Data Shows Lawyers Are Responsible for Many of the Errors." Business Insider, 27 May 2026, https://www.businessinsider.com/increasing-ai-hallucinations-fake-citations-court-records-data-2025-5.

Yin, Yian, et al. "LLM Hallucinations in the Wild: Large-Scale Evidence from Non-Existent Citations." arXiv, 8 May 2026, https://arxiv.org/abs/2605.07723.

 

How Historians Should Evaluate AI

The Historian's AI Evaluation Framework (HAIEF): Twelve Questions Before You Trust a Machine-Written Past

by Alvin Blackshear, retired Historian   <ablackshear@gmail.com>
______________________________________________________________

In the summer of 2026, two of the world's largest consulting firms had to pull back work they had already delivered to clients. PwC withdrew AI-generated reports after discovering they contained fabricated citations and references to sources that did not exist.[1] KPMG ran into the same problem: investigators reviewing one of its AI-assisted reports found that most of its citations were fabricated or misleading.[2] Neither firm works in history. Neither incident involved archives, manuscripts, or a single primary source. That is exactly why they matter here.

It would be easy for historians to read those stories and conclude they describe someone else's problem, a corporate reporting failure, a compliance issue, a cautionary tale about consulting culture. That reading is a mistake. What failed at PwC and KPMG was not domain knowledge. It was sourcing discipline: the basic, mechanical trustworthiness of a citation. If large language models fabricate references in financial and regulatory reporting, where the source material is comparatively narrow and well-indexed, there is no reason to expect better behavior in historical research, where the source base is vast, unevenly digitized, and full of the kind of ambiguity these systems are worst at handling. This is not a story about historians being unusually vulnerable to bad AI output. It is a story about a structural failure mode that will show up wherever citation and provenance matter, and few disciplines depend on citation and provenance more than history does.

There is a second, related problem, and it may be more dangerous than outright fabrication because it is harder to catch. Recent reporting on AI accuracy has noted a troubling pattern: as these systems have become more fluent, they have not necessarily become more honest about what they don't know. Instead, they increasingly deliver incorrect information with the same confident tone as correct information, rather than flagging uncertainty or hedging where hedging is warranted. Any historian will recognize this failure mode immediately, because it is precisely what they are trained to distrust in human sources. A memoir written decades after the fact, a diplomatic cable reporting secondhand rumor as established fact, a witness testifying with total conviction about details that turn out to be wrong — historians have centuries of methodological practice built around the fact that confidence is not evidence. A machine that states falsehoods fluently and without hedging is not a new problem for historical method. It is an old problem wearing a new interface.

Put these two failure modes together, fabricated sourcing and confident error, and you arrive at the central argument of this piece: a language model can perform well on the accuracy benchmarks the AI industry currently uses, and still fail every test a professional historian would apply to a piece of research. General-purpose evaluation metrics measure things like factual accuracy rates, hallucination frequency, and human preference scores. None of those metrics ask whether a source actually exists in the form cited, whether a quotation can be independently located, whether a narrative acknowledges live scholarly disagreement, or whether the chronology holds together under scrutiny. A model can score 95 percent on general factual accuracy and still produce a historical narrative that treats a contested interpretation as settled, leans on secondary literature while ignoring available primary sources, and quietly imports present-day assumptions into a past context that didn't share them. None of that shows up in a hallucination-rate benchmark. All of it would get a paper rejected by a competent peer reviewer.

 That gap, between what general AI evaluation measures and what historical method actually requires, is the subject of this article. This is not a piece about whether historians should use AI, or how to prompt it more effectively; that ground has already been covered well by other writers, including practical workflow guides for AI-assisted historical research. This is a piece about something narrower and, so far, largely unaddressed: how a working historian should grade what the AI hands back. It is meant to function less like an essay and more like a QA protocol, a practical checklist you could actually run against a piece of AI-generated research before you decide whether to trust, cite, or build on it. 

What the research actually shows

The clearest empirical precedent for this kind of evaluation comes from a 2026 comparative study that pitted trained historians against generative AI tools: ChatGPT-4, Claude, Gemini, and Perplexity, on parallel case studies in ancient history. The researchers found a consistent pattern: AI tools were genuinely strong at broad content discovery and thematic synthesis, the kind of work that benefits from fast pattern-matching across large bodies of text. But they were consistently weaker than human historians on source verification, chronological reasoning, provenance, and contextual interpretation, which are the parts of the discipline that depend on judgment rather than retrieval. The study's authors argued for a human-in-the-loop approach that treats AI as a research accelerant rather than a substitute for scholarly adjudication.

A separate line of research has approached the problem from the model side rather than the historian's side, building benchmarks specifically designed to test historical reasoning. That work has found that even the strongest available models struggle with the kind of sophisticated evidentiary reasoning that is routine for a trained historian, the weighing conflicting accounts, recognizing genre and authorial bias, and reconstructing plausible sequences of events from incomplete evidence. This is a useful corrective to the assumption that model capability will simply catch up with historical judgment as systems improve. The gap is not just about scale; it is about a kind of reasoning these systems are not currently built to do well.

Not every recent contribution to this literature is critical of AI's role in historical work. Case studies of experienced historians integrating AI into their actual research workflows, mapping specific tasks to language models, to conceptual work the historian retains, and to conventional computational tools, that describe an approach where AI expands capacity without replacing judgment, provided the historian maintains rigorous cross-checking at every stage. That distinction matters for this piece: the argument here is not that AI is useless for historical work. It's that historians need a way to grade the output that general AI evaluation frameworks were never designed to provide. 

Introducing the framework: HAIEF[3]

What follows is a two-part tool. The first is a practitioner-facing checklist, including twelve diagnostic questions meant to be run quickly against any piece of AI-generated historical research, the way you'd run a gut-check before deciding whether something is worth building on. The second is a more formal, citable rubric that breaks the same underlying concerns into eight scored criteria, useful when you need to document *why* a piece of AI output was accepted or rejected, not just whether it was.

The twelve questions:

1. Are all citations real?

2. Can every quotation be independently located?

3. Is the chronology internally consistent?

4. Does the AI distinguish primary from secondary sources?

5. Are archival collections correctly identified?

6. Are names, dates, and places cross-checked?

7. Does the narrative reflect historiographical debate?

8. Does the AI acknowledge uncertainty?

9. Does it introduce present-day assumptions into historical contexts?

10. Can another historian reproduce the research?

11. Which claims require archival verification?

12. What percentage of the output would I actually trust without checking?

That last question is deliberately blunt, and it's the one I'd argue matters most in practice. Every other question on the list is diagnostic.  It tells you where a piece of AI research is likely to break down. Question twelve forces a number out of you, an honest estimate of how much of the output you'd actually stand behind without independent verification. In my own use, that number is rarely above 40 or 50 percent for anything involving primary-source claims, and treating that as a normal, expected ceiling, rather than a failure of the tool, is the right posture to bring to this work.

The formal rubric maps the same territory onto eight criteria: citation authenticity (do the cited works exist and actually support the claim made), provenance (is every factual assertion traceable to a reliable source), chronological accuracy (do the dates and sequences hold together internally), primary-source fidelity (are quotations and archival references accurate to the original), historiography (does the output acknowledge that scholarly consensus is contested where it is contested), context (are events read within their own historical setting rather than through present-day assumptions), uncertainty (does the output distinguish established fact from inference), and reproducibility (could another historian, working from the same cited evidence, independently reach the same conclusions).

The value of running both versions side by side is that they serve different moments in the workflow. The twelve questions are what you ask while you're reading the output, in real time, deciding whether to keep going or start over. The eight-criterion rubric is what you fill out afterward, when you need a defensible record of why a source was trusted or discarded, which is useful for peer review, for training research assistants, or for documenting due diligence in a professional or editorial context. 

Who this is for

Historians and academic researchers are the most obvious audience, and the stakes for them are the highest: a fabricated citation or an unacknowledged historiographical dispute that makes it into a submitted paper is a professional liability, not just an inconvenience.

Genealogists are arguably the highest-volume, lowest-scrutiny users of AI-assisted historical research, and they are almost entirely unaddressed by the academic literature on this problem. Family history research routinely involves exactly the kind of claims, be it names, dates, places, relationships, that AI systems handle with the least reliability, and genealogists are less likely than academic historians to have institutional habits of source verification already built in.

Journalists covering historical topics under deadline face a particular version of this problem: the twelve-question checklist doubles as a fast fact-checking pass when there isn't time for a full archival dig, and question twelve, what percentage would you trust without checking, is a useful discipline for anyone filing on a clock.

Educators like myself have perhaps the most durable use case. Detecting AI-generated text is a losing, ever-shifting battle. Teaching students to interrogate AI output using a framework like this one, and not asking "did a machine write this" but rather, "would this survive a historian's scrutiny", is a skill that remains useful regardless of how the underlying technology changes. 

Closing

None of this should be mistaken for a claim that a checklist can do a historian's job. A framework like HAIEF can catch a fabricated citation, flag an inconsistent chronology, or surface a claim that needs archival verification before it gets treated as settled. What it cannot do is listen. Jan Burzlaff has argued that AI systems summarize but do not listen, reproduce but do not interpret.  These tools  achieve coherence while faltering at contradiction, and that historical writing is not simply an output to be optimized but a form of presence, a risk taken in the act of trying to make meaning where no prewritten frame will do. That distinction is worth sitting with rather than resolving. A verification framework can tell you whether the sourcing holds up. It cannot tell you whether the history has been understood. Those are different questions, and confusing them, mistaking a clean citation audit for genuine historical interpretation, may turn out to be the more consequential failure mode of all.

Sources

Burzlaff, Jan. "Fragments Not Prompts: Five Principles for Writing History in the Age of AI." Rethinking History (2026). https://doi.org/10.1080/13642529.2025.2546174.

Campbell, Chris. "The Historian in the Age of AI." Transactions of the Royal Historical Society (December 10, 2025). https://www.cambridge.org/core/journals/transactions-of-the-royal-historical-society/article/historian-in-the-age-of-ai/37E3B742A2983DF4DAA2E38D48252F89.

"Generative AI as a Historical Source: Source Criticism, Citation Integrity, and the Jagged Frontier of Digital History." (March 20, 2026). https://acnsci.org/journal/index.php/cte/article/view/1438.

Fox, Yaniv. Using Generative AI in Historical Practice. Cambridge Elements. Cambridge: Cambridge University Press, 2026. https://www.cambridge.org/core/elements/abs/using-generative-ai-in-historical-practice/7C1392A6E9DBD379FAA42E2D16A6D45B.

Henriot, Christian. "The AI-Augmented Research Process: A Historian's Perspective." Preprint, arXiv, August 3, 2025. https://arxiv.org/abs/2508.01779.

"HistoRAG: Embedding Historical Methodology in Retrieval-Augmented Generation." Preprint, arXiv, June 16, 2026. https://arxiv.org/abs/2606.18103.

"Can LLMs Act as Historians?" Preprint, arXiv, April 27, 2026. https://arxiv.org/abs/2604.24690.

Ruchniewicz, Krzysztof. "AI in History Education: The American Historical Association's Guidelines, the Views of Their Authors, and Their Reception within the Historical Community." Studia Historica 62, no. 1 (2026). https://doi.org/10.14746/sh.2026.62.1.003.

Solga, Raymond S., and Mohammed J. Sarwar. "Evaluating Generative AI in Historical Research: A Comparative Study on Identifying Primary Source Evidence in Ancient History." AI & Antiquity 2, no. 1 (February 26, 2026). https://doi.org/10.64946/aiantiquity.v2i1.003.

Financial Times. "PwC Withdraws AI-Generated Reports over Fabricated Citations." July 29, 2026. https://www.ft.com/content/7e149ac8-2ce2-4266-8940-192f9821b33c.

TechRadar Pro. "A Major KPMG Report on AI Was Found to Be Chock-Full of AI Hallucinations." June 12, 2026. https://www.techradar.com/pro/a-major-kpmg-report-on-ai-was-found-to-be-chock-full-of-ai-hallucinations.

Axios. "AI Chatbots Are Getting More Confident — and Sometimes More Wrong." May 30, 2026. https://www.axios.com/2026/05/30/ai-accuracy-chatbots-hallucinations.

---

[1] https://www.ft.com/content/7e149ac8-2ce2-4266-8940-192f9821b33c (2026-07-29)

[2] https://www.techradar.com/pro/a-major-kpmg-report-on-ai-was-found-to-be-chock-full-of-ai-hallucinations (2026-06-12)

[3] HAIEF = Historian's AI Evaluation Framework