Monday, August 17, 2026

The Nicholas Brothers and the Shadow of Minstrelsy

 

The "Jumpin' Jive" sequence from Stormy Weather (1943), featuring Cab Calloway and his orchestra alongside tap dancers Fayard and Harold Nicholas, remains one of the most widely praised dance sequences ever filmed. Calloway performs the song with his band while the Nicholas Brothers erupt into a routine that carries them across music stands, atop a piano, and down a long staircase, ending in a series of splits. According to the brothers' own later account, the number was never rehearsed and was captured in a single take.

The admiration for it is genuinely raising a complicated question. The choreography itself is not a minstrel routine. It draws on tap's roots in vaudeville and jazz-era "flash dancing," an acrobatic style the brothers helped define. But the number exists inside a film, and an industry, still shaped by the same minstrel tradition discussed elsewhere in this project. Stormy Weather was one of the only major studio productions of its era built around a Black cast, at a time when Black performers were routinely denied lead roles elsewhere. Even so, the Nicholas Brothers' own dance sequences were frequently structured as self-contained numbers, detachable from the plot, so that theaters in the segregated South could cut them from prints entirely. The brilliance of the performance and the racist constraints surrounding its production existed at the same time, not as a contradiction to explain away but as the actual condition under which the work was made.

Fred Astaire's admiration for the brothers is well documented, and it came from someone with no professional incentive to overstate a rival's skill. After watching the "Jumpin' Jive" number, Astaire is widely reported to have called it the greatest dancing he had ever seen on film. Other dancers made similar claims about the brothers across their careers, including Mikhail Baryshnikov and Gregory Hines. Astaire's own signature style, on display a few years earlier in Top Hat (1935), was built around elegance, restraint, and close partnership choreography with Ginger Rogers, a very different vocabulary from the Nicholas Brothers' acrobatic "flash" tradition. That two dancers working in such different registers could each be called the best of their era says something about how wide tap's range actually was, and how much of that range was shaped by Black innovators who rarely received equivalent billing or resources.

None of this resolves neatly. A viewer can recognize the "Jumpin' Jive" sequence as one of the great achievements in American dance and still notice that it appears inside a system that limited how, when, and where Black performers could be seen. The tension is not a flaw in the work. It is the historical fact the work was made under, and it is part of what makes watching it now, nearly eighty years later, more than simple entertainment.

Sources

Posted by at 9:51 AM 0 comments

Blackface in History and Contemporary Comedy


by Alvin Blackshear  |  Historian & Researcher  <ablackshear@gmail.com>

Racism in American entertainment rarely announces itself as a single, settled question. It shows up instead as a set of overlapping stories, each one complicating the last. The history of blackface, and its strange half-life in comedy today, is one of the clearest examples of that complexity.

Ted Danson, 1993

On October 8, 1993, Ted Danson appeared at a New York Friars Club roast honoring Whoopi Goldberg, with whom he was then romantically involved. He performed in blackface makeup with exaggerated white lips and repeatedly used a racial slur during a set built around jokes about their relationship. Contemporary reporting from the event describes Danson wearing blackface makeup and large white lips while using the slur several times and joking about his sex life with Goldberg.

The response was immediate and sharply divided. Then-Mayor David Dinkins and other attendees objected to the material, and Montel Williams walked out and resigned from the club, writing in his resignation telegram that he had been confused about whether he was at a Friars event or a rally for the Klan and Aryan Nation. Goldberg, by contrast, defended Danson publicly, telling reporters that the two of them had written the material together and were not trying to be politically correct but to be funny for themselves.

That defense is the complicating detail people tend to forget. It was not simply a white performer testing a racist trope. It was a Black performer with a personal stake in the material and, according to her own account, a hand in creating it. The moral clarity of "blackface is bad" runs into real friction when the collaborator defending it is the person the routine was ostensibly honoring.

More than three decades later, Danson revisited the incident himself. In June 2026, he told podcast host W. Kamau Bell that he intended to keep apologizing for the rest of his life, and rejected the idea that his intentions at the time should count for much, saying the impact on people is what matters, not the intent behind it. He also described how the footage resurfaced during the Black Lives Matter movement and cost him work, and credited conversations facilitated by Jane Fonda and author Heather McGhee with helping him understand what he had done. That a public figure would spend over thirty years working through, rather than simply outliving, an old controversy suggests these reckonings are rarely as fast or as final as news cycles imply.

The Kake Walk at the University of Vermont

A very different kind of blackface tradition took shape at the University of Vermont beginning in the early 1890s. The Kake Walk grew out of the American cakewalk and minstrel traditions and eventually became a formal fraternity competition, with pairs of male students performing exaggerated high-stepping dances to a tune called Cotton Babes, often alongside minstrel-style skits. By the mid-twentieth century it had become the centerpiece of UVM's Winter Carnival, drawing crowds that reportedly reached around eight thousand students, alumni, and local residents at its peak in the Patrick Gym.

The instructive part of this history is that opposition did not begin in the 1960s, as is often assumed. Historians James Loewen and Larry McCrorey traced organized dissent back to 1954, when the fraternity Phi Sigma Delta refused to perform in blackface, a decision that drew criticism from the wider campus community. Objections continued to appear in the student newspaper and local press for years afterward.

The university's actual path away from the tradition was gradual and, in retrospect, uncomfortable in its own right. In 1963 the Interfraternity Council voted to replace blackface with light green makeup while keeping the exaggerated dialect intact. After audience complaints that the green was not dark enough, the makeup was changed to a darker green in 1964, a color UVM's own archival materials note is often indistinguishable from blackface in black-and-white photographs. The Kake Walk was not formally eliminated until 1969, following decisions by a student-faculty committee, the Student Association, and the Interfraternity Council, over the strong objections of alumni who considered it a cherished tradition.

There is a further layer worth preserving rather than flattening. The cakewalk itself originated among enslaved African Americans, and was later appropriated by white minstrelsy and used to reinforce racial hierarchy after Reconstruction. UVM's Kake Walk was a specific collegiate distortion of that longer, more complicated Black history, not a synonym for it. Collapsing the two stories loses something real about how appropriation works.

Black comedians and the same imagery, used differently

Blackface's afterlife in comedy has also been shaped by Black performers testing its logic rather than repeating it. Dave Chappelle has used blackface repeatedly as a reference point for thinking about where representation becomes harmful. In his 2021 Netflix special The Closer, he compared identifying as transgender to the act of wearing blackface, a comparison that became one focus of the criticism and employee walkout that followed the special's release. Netflix employees organized a walkout at the company's Los Angeles offices, and executives, including co-CEO Ted Sarandos, later acknowledged mishandling their initial response to staff concerns.

Chappelle's discomfort with his own earlier work followed a related logic. He has described growing uneasy with the racial humor on Chappelle's Show not because of what he intended a sketch to mean, but because he could not control whether audiences were laughing at racism or simply enjoying the stereotype itself.

Key and Peele approached the same territory from the opposite direction. In the 2012 sketch "Das Negros," the pair play two Black men in Nazi-occupied Germany who disguise themselves in whiteface to evade a Nazi officer searching for Black people, a premise built entirely around exaggerating how visibly the disguise fails. The sketch uses racial makeup as a tool of satire aimed outward at the absurdity of racial policing, rather than as a costume worn for its own sake.

What these cases share

Taken together, these episodes resist a single tidy lesson. A white actor's blackface routine was defended, in the moment, by the Black woman it was framed around. A university's opposition to a blackface tradition predates the era most people associate with civil rights activism, and its actual abolition took the form of a compromise, green makeup, that was arguably almost as troubling as what it replaced. And Black comedians today are still testing, from different directions, whether racial imagery can be turned into a critique of racism rather than a restatement of it. The common thread is not a rule but a question that keeps resurfacing: whether an audience is laughing at the target of a joke or at the harm the joke depicts, and how much control a performer actually has over which one happens.

References

Ted Danson and the 1993 Friars Club roast

The Kake Walk at the University of Vermont

Dave Chappelle, The Closer, and the Netflix walkout

Key & Peele, "Das Negros"

Suggestions for Further Reading and Research

For readers who want to go deeper than a single article can, a few directions are worth pursuing:

On the history and mechanics of blackface minstrelsy. Eric Lott's Love and Theft: Blackface Minstrelsy and the American Working Class remains a standard scholarly starting point for understanding how minstrelsy functioned as both racist spectacle and, paradoxically, a form of cross-racial cultural exchange. Spike Lee's 2000 film Bamboozled is a useful fictional companion piece, dramatizing many of the same tensions around audience reception that appear in the Chappelle discussion above.

On the Kake Walk specifically. The essay collection Darkology: Blackface and the American Way of Entertainment (Liveright) includes a chapter, "Sugarcoating Slavery: How the University of Vermont's Kake Walk Made Blackface a Collegiate Sport," that goes well beyond what a short article can cover. UVM's own digital collections are unusually rich for a case study like this, including alumni correspondence objecting to the tradition's end, which is worth reading directly rather than through summary.

On the cakewalk's origins before appropriation. Because this article treats the Kake Walk and the original cakewalk as related but distinct, readers interested in the earlier, African American history of the cakewalk itself may want to look into its documented origins on plantations in the antebellum South and its subsequent life in Black vaudeville, which is a separate and much larger story than the UVM adaptation.

On satire, intent, and audience reception in comedy. The recurring question in this article, whether a joke critiques a stereotype or simply reproduces it for an audience that enjoys the stereotype, is a long-running debate in comedy criticism. Readers might compare how this question has been raised around other satirists who use identity-based material, including Norman Lear's All in the Family and its handling of Archie Bunker's bigotry, as an earlier television-era version of the same argument.

On contemporary accountability and apology. Heather McGhee's The Sum of Us: What Racism Costs Everyone and How We Can Prosper Together, cited by Danson himself as influential in his own reflection, offers a broader economic and social argument about the costs of racism that extends well past entertainment and into policy.

Posted by at 9:11 AM 0 comments
Labels: ,

Wednesday, August 12, 2026

The Biggest Mistake Historians Make When Using ChatGPT

 

by Alvin Blackshear  |  Historian & Researcher

A historian asks ChatGPT a seemingly straightforward question: “Who was the first African American elected to public office in Delaware after Reconstruction?”

Within seconds, the system produces a polished answer. It supplies a name, an election date, a description of the office, and perhaps even several citations. The prose is orderly, confident, and specific. Nothing about it appears careless. The historian copies the response into a research notebook and proceeds to the next question.

Only later does the answer begin to unravel. One citation does not exist. A quotation attributed to a nineteenth-century newspaper is actually a modern paraphrase. The date belongs to another officeholder. An appointment has been confused with an election. Details from two people have been combined into a single biography.

Nothing in the original response sounded implausible. That is precisely the danger.

The greatest mistake historians make when using ChatGPT is not merely trusting an occasional incorrect answer. It is treating AI-generated prose as though it were historical evidence. ChatGPT produces narratives. Historians require evidence. Those activities may sometimes overlap, but they are not the same.

The Abandonment of Historical Method

Historians already possess a sophisticated method for evaluating claims. When examining a letter, newspaper, memoir, photograph, government record, or oral history, they ask familiar questions. Who created the source? When was it created? For what purpose? Under what circumstances? What interests shaped its contents? Can its claims be corroborated? Has another scholar interpreted the evidence differently?

These questions constitute more than an academic ritual. They are the means by which historical knowledge is distinguished from rumor, memory, advocacy, folklore, and invention.

Yet something curious happens when many researchers begin using generative artificial intelligence. The normal habits of source criticism are suspended. A response is accepted because it sounds reasonable. A citation is trusted because it resembles an academic citation. A quotation is repeated because the language seems appropriate to the period.

The historian who would never accept an unsigned reminiscence without examining its provenance may accept an AI-generated paragraph without asking where any of its claims originated.

ChatGPT should therefore be treated as an intelligent research assistant, not as a documentary source. It can suggest search terms, identify possible lines of inquiry, compare concepts, organize notes, formulate questions, and help clarify difficult prose. It can also point a researcher toward archives, books, people, and events that merit investigation. Its usefulness, however, does not convert its output into evidence.

The distinction is fundamental. A research assistant may tell a historian that a document probably exists. The historian must still locate and examine it.

Why Fluency Is So Persuasive

Large language models are especially dangerous when they are almost correct. An obviously absurd answer invites skepticism. A plausible answer, containing real names, accurate background information, and one fabricated detail, may pass unnoticed.

ChatGPT’s authority is largely rhetorical. It writes in complete sentences, arranges events chronologically, supplies transitions, and often presents conclusions without visible hesitation. Readers naturally associate these qualities with knowledge. Confidence, specificity, and coherence become substitutes for provenance.

Recent research suggests that this problem is not simply the result of careless prompting. Kalai and his colleagues argue that common accuracy-based evaluations may reward models for guessing instead of acknowledging uncertainty. If a benchmark gives credit only for correct answers and imposes no meaningful cost for plausible errors, the model is encouraged to answer even when abstention would be more responsible. The researchers propose open scoring rubrics that disclose how errors and abstentions will be evaluated.

This observation has important consequences for historians. A model that always produces an answer may appear more useful than one that frequently says, “I do not have enough evidence.” In historical research, however, an admission of uncertainty may be the more accurate and intellectually responsible response.

The historian must therefore evaluate not only what the system says, but whether the system should have answered at all.

Five Forms of Historical Hallucination

Historical hallucinations frequently assume recognizable forms.

The first is the invented citation. ChatGPT may produce a realistic book title, journal article, archival collection, author, volume number, or page range. Every component may appear academically credible even though the source does not exist.

The second is the misquotation of a primary source. A model may modernize, compress, or reconstruct a statement and then present the resulting language inside quotation marks. The general sentiment may be accurate, but the words are not documentary evidence.

The third is the composite biography. Details belonging to several individuals may be joined into one apparently coherent life. This is especially dangerous when researching people who share names, occupations, institutions, military units, or geographic locations.

The fourth is incorrect chronology. A model may identify real events but arrange them in the wrong order, confuse the date of an appointment with the date of an election, or place a person at an institution before that person arrived there.

The fifth is false causation. ChatGPT may connect two events with phrases such as “therefore,” “as a result,” or “this led to,” even when the evidence establishes only sequence or correlation.

These errors are not equally visible. An invented person may be discovered quickly. A subtle chronological error or unsupported causal inference may survive several rounds of editing because it fits an expected narrative.

Why Better Prompts Are Not Enough

Prompt design can improve an AI response. A historian can request citations, ask the system to distinguish facts from interpretations, require expressions of uncertainty, or instruct it not to invent missing information. Such practices are worthwhile.

They are not verification.

A carefully written prompt may reduce the probability of an error, but it cannot transform generated language into historical evidence. Even a response that includes citations must be checked against the cited material. Even a quotation accompanied by a page number must be located on that page. Even a claim presented as certain must be corroborated.

Research on hallucinations in academic writing identifies fabricated citations, factual distortion, named-entity errors, logical inconsistency, and propagation errors as threats to scholarly integrity. The final responsibility remains with the human author. An historian cannot excuse a false statement by explaining that ChatGPT supplied it.

Better prompting is therefore a research technique, not an evidentiary standard.

Testing AI as Historians Test Evidence

Historians need a repeatable method for testing AI-generated research. The evaluation should move beyond impressions such as “the answer seemed good” or “the model performed well.”

A practical scoring rubric might assess eight dimensions:

1.      Citation existence: Does every cited work or archival collection exist?

2.      Citation accuracy: Does the cited source support the specific claim?

3.      Quotation fidelity: Are quoted words reproduced exactly and in context?

4.      Identity accuracy: Have people with similar names or careers been distinguished?

5.      Chronological accuracy: Are dates and sequences correct?

6.      Geographic and institutional accuracy: Are places, offices, organizations, and jurisdictions correctly identified?

7.      Provenance: Can each important assertion be traced to accessible evidence?

8.      Historiographical consistency: Does the answer acknowledge significant scholarly disagreement?

Each category could be scored from zero to four. A zero would indicate fabrication or complete failure. A score of one would reflect major errors. Two would indicate partial support or substantial ambiguity. Three would represent generally accurate work with minor deficiencies. Four would require full and independently verified support.

A weighted rubric would assign greater penalties to invented citations, false quotations, and merged identities than to minor stylistic or contextual omissions. This matters because not all errors have the same scholarly consequence.

Benchmark testing should also use a fixed set of questions rather than memorable anecdotes. Researchers could create a collection of historical questions at several difficulty levels: well-documented national events, obscure local events, disputed interpretations, biographical identity problems, chronological puzzles, and questions whose correct answer is genuinely unknown.

The same questions could then be submitted to several AI systems under controlled conditions. Evaluators would record accuracy, citation validity, unsupported assertions, appropriate abstentions, and changes across repeated runs. Such testing would reveal not only whether a model produces correct answers, but how it fails.

What RAG Can and Cannot Do

Retrieval-augmented generation, commonly known as RAG, offers one method of improving reliability. A RAG system retrieves documents from a designated collection before generating an answer. For historians, that collection might contain archival finding aids, digitized newspapers, oral-history transcripts, government records, or scholarly articles.

This approach can make AI responses more transparent and more closely connected to identifiable sources. It may reduce fabricated citations and permit researchers to inspect the evidence used in producing an answer.

Yet RAG does not eliminate the need for historical judgment.

Recent research distinguishes factuality from faithfulness. A statement may be faithful to a retrieved document while the document itself is inaccurate, biased, incomplete, or misinterpreted. Conversely, a statement may be factually correct but unsupported by the particular documents retrieved. A serious evaluation must test both dimensions.

For historians, retrieval is only the beginning. The fact that a system retrieved a newspaper article does not establish that the article was truthful. The fact that a claim appears in a memoir does not establish that memory was reliable. The fact that several sources repeat a statement does not prove they were independent.

RAG can retrieve evidence. It cannot perform the full intellectual work of source criticism.

The Historian’s Continuing Responsibility

ChatGPT does not threaten historical scholarship simply because it occasionally invents facts. Historical scholarship has always confronted error, forgery, propaganda, partial memory, and misleading testimony.

The greater danger arises when historians allow the fluency of artificial intelligence to replace the discipline’s oldest habit: skepticism.

For centuries, historians have questioned manuscripts, memoirs, newspapers, photographs, statistics, and archives. Artificial intelligence deserves no exemption from that tradition. Its output should be tested, scored, corroborated, and traced to evidence.

The historian’s task remains unchanged. It is not merely to discover information or compose a persuasive narrative. It is to determine what the surviving evidence permits us to say, what remains uncertain, and why the distinction matters.

References

  1. Kalai, Adam Tauman, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. “Evaluating Large Language Models for Accuracy Incentivizes Hallucinations.” Nature 653 (2026): 1047–1051. https://www.nature.com/articles/s41586-026-10549-w
  2. Rahman, Subhey Sadi, et al. “Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models.” Artificial Intelligence Review 59 (2026), article  70. https://link.springer.com/article/10.1007/s10462-025-11454-w
  3. Fadeeva, Ekaterina, et al. “Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval-Augmented Generation.” Findings of the Association for Computational Linguistics: ACL 2026 (2026): 6814–6836. https://aclanthology.org/2026.findings-acl.338
  4. Giray, Louie, Md Emdadul Islam, and John Arvin Glo. “AI Hallucinations in Academic Writing: Implications for Research Integrity.” Naunyn-Schmiedeberg’s Archives of Pharmacology (2026). https://pubmed.ncbi.nlm.nih.gov/42217043 
Posted by at 4:44 PM 0 comments
Labels: , , , , , ,

Monday, August 10, 2026

A Rubric for Assessing AI-Generated FOIA Summaries

by Alvin Blackshear | Historian & Researcher


Intro: A Gap Hiding in Plain Sight

Two literatures are growing quickly in 2026 and, so far, growing apart. The first is the literature on large language model (LLM) evaluation: rubric-driven frameworks for scoring factuality, faithfulness, hallucination, and task completion across generative outputs. The second is the literature on FOIA (Freedom of Information Act) automation: case studies and vendor analyses describing how agencies are using AI to triage requests, search collections, and draft redactions faster than manual review allows. Both bodies of work are maturing on their own terms. Neither has, in any formal sense, met the other.

That absence is notable given the setting. FOIA is not an ordinary text-summarization domain. A FOIA summary sits downstream of a legal disclosure decision. It distills a record that has already been searched, reviewed, exempted, and redacted, and it typically becomes part of what a requester, a court, an oversight body, or a journalist relies on to understand what an agency did and why. If an AI-generated summary drops a redacted date, blurs the line between what was withheld and what was released, or introduces even a plausible-sounding inference not supported by the underlying record, the error does not stay contained to a chatbot transcript. It can shape a public record, an appeal, or a legal filing.

Despite that stakes profile, a search of the current literature turns up remarkably little that combines the two threads. Papers on LLM evaluation rubrics rarely mention FOIA. Papers on FOIA automation rarely propose a scoring framework specific to summary quality. Almost none attempt a rubric purpose-built for the FOIA summarization task, one that treats redaction fidelity, exemption logic, and traceability back to the released record as first-class evaluation criteria, on par with the factuality and hallucination metrics borrowed from general-purpose LLM evaluation. That gap is the occasion for this article: a proposed rubric for assessing AI-generated FOIA summaries, and a look at how the leading purpose-built FOIA technology on the market today, Relativity FOIA and Relativity aiR, fits into, and exposes, that same gap.

Why FOIA Summarization Needs Its Own Rubric

General-purpose summarization rubrics, the ones built for news articles, meeting transcripts, or customer support tickets, optimize for coherence, conciseness, and topical coverage. Those criteria matter for FOIA summaries too, but they are not sufficient. A FOIA summary carries legal weight that a meeting-notes summary does not. Three features of the FOIA context justify a dedicated evaluation instrument:

·         Redaction is structural, not incidental. A FOIA record is rarely handed over whole. Exemptions under the statute (and analogous state public-records laws) remove specific categories of information, such as personal privacy, law enforcement techniques, or deliberative process, while leaving the rest disclosable. A summary that fails to distinguish "this is what was released" from "this is what was withheld and why" misrepresents the disclosure itself, not just the document.

·         Legal meaning is precise and often reader-facing. Terms like "responsive," "exempt," "segregable," and the specific statutory exemption cited (e.g., Exemption 5, Exemption 7(C)) carry defined legal consequences. A summary that paraphrases those terms loosely can change what a reader believes the agency determined.

·         The audience includes adversarial and oversight readers. Unlike an internal meeting summary, a FOIA summary may be read by litigators, journalists, or Inspectors General specifically looking for inconsistency between the summary, the redactions, and the underlying record. The evaluation bar has to anticipate that scrutiny.

These features are why importing an off-the-shelf LLM summarization rubric is inadequate on its own, and why FOIA-specific criteria, including redaction and exemption handling, legal-meaning preservation, and traceability to the released record, need to sit alongside more familiar accuracy and hallucination metrics.

A Proposed Rubric for AI-Generated FOIA Summaries

The rubric below is offered as a starting framework rather than a finished instrument. It borrows its general shape, weighted criteria that are each independently scorable, from the structured evaluation approaches now common in LLM benchmarking, while grounding the specific criteria in FOIA practice.

 Criterion

Weight

 Factual accuracy

25%

 Completeness of disclosed information

20%

 Correct treatment of redactions and exemptions

15%

 Preservation of legal meaning

15%

 Hallucination rate

10%

 Neutrality and absence of editorial bias

5%

 Citation/traceability back to released records

5%

 Readability and organization

5%


Factual accuracy (25%) carries the largest weight because it is the load-bearing criterion for every downstream use of the summary. Evaluators should check whether every factual claim in the summary is verifiable against the record as released, with particular attention to dates, names of programs or offices (as distinct from names withheld under privacy exemptions), dollar figures, and sequences of events.

Completeness of disclosed information (20%) asks whether the summary captures the material substance of what was released, not merely a plausible-sounding excerpt of it. A summary that is accurate but omits a disclosed finding, a key admission, or a significant data point can be just as misleading as one that fabricates something, because it gives a false impression of a "clean" record.

Correct treatment of redactions and exemptions (15%) is the most FOIA-specific criterion on the list. It asks whether the summary clearly signals where information was withheld, whether it correctly reflects the cited exemption basis where that basis is itself disclosed, and, critically, whether it avoids inferring or reconstructing withheld content from context. That kind of inference is a known failure mode of generative summarization when a document has gaps.

Preservation of legal meaning (15%) evaluates whether statutory and procedural terms are used correctly and whether the summary's characterization of the agency's determination (responsive, non-responsive, partially exempt, referred to another agency, etc.) matches the record's actual disposition.

Hallucination rate (10%) is scored separately from general factual accuracy because it targets a distinct failure mode: content that has no basis anywhere in the source record, rather than content that is merely imprecise. In FOIA summarization this includes fabricated exemption rationales, invented dates, or synthesized "likely" content behind a redaction.

Neutrality and absence of editorial bias (5%) checks that the summary does not editorialize about the agency's conduct, insert framing not present in the record, or adopt a tone that implies a conclusion (such as wrongdoing, cover-up, or exoneration) the record itself does not support.

Citation/traceability back to released records (5%) asks whether a reviewer can trace each summary statement back to a specific page, Bates number, or section of the released record. This matters enormously for defensibility if the summary is later challenged.

Readability and organization (5%) is weighted lowest deliberately. It matters for usability, but a highly readable summary that fails the criteria above is a worse outcome than an awkward one that passes them.

Applied consistently, a rubric of this kind would let researchers and agencies compare AI-generated FOIA summaries across vendors, models, and prompt designs using a shared, FOIA-native standard, something the current literature does not yet offer.

Where the Industry Stands: Relativity FOIA and Relativity aiR

Relativity is a useful lens for testing this gap against real-world deployment, because it is one of the few vendors building AI specifically into the FOIA review workflow rather than adapting a general eDiscovery tool after the fact.

Relativity FOIA, announced as generally available inside RelativityOne Government, unifies records request intake, case management, AI-supported review, disclosure, and reporting into a single connected workflow. The product's own framing makes clear that AI is meant to assist rather than replace the reviewer: every prioritized record arrives with a confidence score, a plain-language explanation, a counterpoint, and supporting citations, and it is the human reviewer, not the model, who makes the final call on each document. The accompanying launch announcement strikes the same note, describing the platform as pairing defensible AI and automation with Relativity's established review technology so that agencies can standardize exemption rationale and accelerate review. That is a framing built around assistance and consistency, not autonomous disclosure decisions.

Relativity aiR, the generative AI layer underlying that workflow, does not market a standalone "summarization" feature by that name, but its analysis outputs function as summaries in practice. RelativityOne's government release notes describe a mid-2026 update to aiR for Review that lets teams build custom analyses across both text and images, producing what the release notes themselves describe as "insights, extractions, and summaries." In other words, the document-level explanations aiR for Review generates, the kind of output a FOIA reviewer would read to understand a record before deciding what to release, are a summarization capability in every functional sense, even without that specific product label.

The philosophy behind these design choices is spelled out most directly in Brian Thompson's commentary on government AI infrastructure. Thompson argues that courts and oversight bodies require agency decisions to be explainable, traceable, and defensible, and that AI can only meet that bar when it is purpose-built for legal and public sector work in a way that preserves audit trails and keeps a verifiable link between inputs and outputs, with human validation built in rather than bolted on. That is a strong statement of principle: transparency, defensibility, explainability, and reproducibility as design requirements. But it remains a statement of principle rather than a published measurement framework.

The clearest evidence of both the industry's evaluation ambition and its current limits comes from an independent study Relativity has publicized: Redgrave LLP's head-to-head comparison of aiR for Review against a traditional active-learning managed review. The results were striking on the metrics the study did measure. aiR for Review reached 88 percent recall against 64 percent for the active-learning workflow, and its elusion rate (responsive documents wrongly left in the discard pile) came in at 1 percent versus 3 percent for the manual process, while consuming roughly 18 attorney-hours against an estimated 1,123 hours for the 24-person manual review team. The study is equally candid about tradeoffs: aiR for Review's precision, at 29 percent, trailed the active-learning workflow's 39 percent, a gap the authors attribute in part to the low "richness," meaning the scarcity of responsive documents, in the test population.

That study, however, measured a responsiveness-classification task, finding documents relevant to a legal standard, not a summarization task. It offers precision and recall figures for document identification, not for the fidelity of AI-generated summaries. Reviewed against the rubric above and against the broader FOIA-specific commentary, six gaps stand out in Relativity's current public-facing material:

·         There is no formal, published FOIA summarization evaluation rubric comparable to the one proposed here.

·         There are no benchmark scores specific to AI-generated FOIA summaries, as distinct from the responsiveness-classification benchmarks that do exist.

·         There is no hallucination testing specific to FOIA summaries. The closest published work addresses document-level responsiveness accuracy, not summary-level fidelity.

·         There is no disclosed human evaluation methodology specific to FOIA summarization, meaning no description of how human reviewers score AI summary outputs against a defined standard.

·         There are no precision/recall studies of AI-generated summaries themselves, again as distinct from precision/recall for document classification.

·         There is no academic validation paper describing how Relativity measures FOIA summary quality specifically, as opposed to review-workflow efficiency more broadly.

None of this is a criticism of the product design, which is explicitly built around human validation, citation-backed outputs, and audit trails, features that a rigorous FOIA summarization rubric would actually reward under the "traceability" and "correct treatment of redactions and exemptions" criteria. It is, instead, a description of an open research space. The underlying architecture for defensible AI summarization exists, but the formal instrument for measuring whether it produces good FOIA summaries, as opposed to good responsiveness calls, does not yet exist in the public record.

Implications for Research and Practice

The rubric proposed here is deliberately modest in scope: eight criteria, weighted to reflect FOIA's legal stakes, designed to be usable by researchers, agency FOIA officers, and vendors alike. Its value lies less in the specific weights, which should be stress-tested and adjusted through empirical study, than in establishing that FOIA summarization deserves a dedicated evaluation standard rather than an inherited one.

For researchers, the immediate opportunity is to pair this kind of rubric with a labeled dataset of AI-generated FOIA summaries scored by trained reviewers, producing the kind of benchmark and hallucination-rate figures currently missing from the literature. For agencies and vendors, the opportunity is to publish exactly the kind of validation study that Relativity has already modeled for document responsiveness (blind expert review, ground-truth comparison, transparent methodology) but aimed squarely at summary quality rather than classification accuracy.

As FOIA request volumes climb and agencies lean further into AI-assisted review, the gap between AI that helps reviewers understand records and AI whose summaries have been rigorously measured against a FOIA-specific standard is the gap this research agenda should close.

Sources

1.     Relativity, "Relativity FOIA: Purpose-Built FOIA Software for Federal and State Agencies," June 2026. https://relativity.com/blog/relativity-foia-purpose-built-foia-software-for-federal-and-state-agencies

2.     Relativity, "Relativity Launches Relativity FOIA to Streamline Public Disclosure Operations for Government Agencies," June 1, 2026. https://relativity.com/news-events/relativity-launches-relativity-foia-to-streamline-public-disclosure-operations-for-government-agencies

3.     Relativity, "What's New in RelativityOne Government (FOIA)." https://help.relativity.com/RelativityOne/Content/What_s_New/What_s_new_in_RelativityOne_Government.htm

4.     Brian Thompson, "AI, Operational Infrastructure, and the Future of Government Legal Work," Relativity Blog, April 8, 2026. https://www.relativity.com/blog/ai-operational-infrastructure-and-the-future-of-government-legal-work

5.     Robert Keeling and Ray Mangum (Redgrave LLP), "Results Are In: 5 Lessons from an Independent Study of aiR for Review," Relativity Blog, June 9, 2026. https://www.relativity.com/blog/results-are-in-5-lessons-from-an-independent-study-of-air-for-review



Posted by at 6:02 AM 0 comments
Labels: , , ,
Subscribe to: Posts (Atom)