A New York attorney cited six cases that didn’t exist. A Colorado lawyer submitted a brief full of fabricated judicial opinions. A Texas firm filed a motion riddled with phantom precedents. These aren’t hypotheticals from a legal ethics seminar. They are real incidents — documented, sanctioned, and increasingly common — that reveal a systemic crisis spreading through American courtrooms at a pace that neither judges nor bar associations seem prepared to handle.
The problem is deceptively simple. Large language models like ChatGPT, Claude, and others can generate text that reads like polished legal prose. It cites cases. It quotes holdings. It constructs arguments with the confident cadence of a seasoned litigator. But it also invents things. Frequently. And when lawyers fail to verify what these systems produce, the results land in front of judges who are, to put it mildly, not amused.
As Futurism reported in a detailed examination of the phenomenon, AI-generated legal filings have created a rolling wave of courtroom chaos that shows no signs of cresting. The publication documented case after case in which attorneys submitted AI-fabricated citations, were caught by opposing counsel or judges, and then faced sanctions, fines, or worse. The pattern is remarkably consistent: a lawyer under time pressure turns to an AI chatbot, receives output that looks authoritative, and files it without checking whether the cited cases are real.
The most infamous example remains the case of Steven Schwartz, a personal injury attorney at the New York firm Levidow, Levidow & Oberman. In 2023, Schwartz used ChatGPT to research a brief filed in federal court in a case against the Colombian airline Avianca. The brief cited six cases. None of them existed. When the presiding judge, P. Kevin Castel of the Southern District of New York, demanded explanations, Schwartz admitted he had used the AI tool and hadn’t verified its output. Judge Castel imposed a $5,000 fine on Schwartz and his colleague Peter LoDuca, calling the fabricated citations an “unprecedented circumstance” and noting that the attorneys had “abandoned their responsibilities” as officers of the court. The case became a national cautionary tale. But it didn’t stop the problem.
Not even close.
In Colorado, attorney Zachariah Crabill was suspended for filing AI-generated motions in family court that contained invented case citations and fabricated quotes attributed to real judges. According to court documents, Crabill used ChatGPT to draft motions in a custody dispute. The fictitious citations weren’t just wrong — they were detailed enough to appear plausible, complete with volume numbers, page references, and holdings that sounded reasonable. The Colorado Supreme Court’s Office of Attorney Regulation Counsel pursued disciplinary action, and Crabill received a suspension. As Futurism noted, the case underscored that the hallucination problem extends well beyond the federal courts and into state-level proceedings where the stakes for individuals — custody of children, in this instance — couldn’t be higher.
The term “hallucination” has become the industry’s preferred euphemism for what these AI systems do. It sounds almost benign, like a minor glitch. It isn’t. When a large language model generates a fabricated case citation, it is producing a lie dressed in the syntax of truth. The model doesn’t know the case is fake. It doesn’t “know” anything. It predicts the next most likely token in a sequence, and legal citations happen to be sequences it can mimic with disturbing fluency. The result is text that can fool not just hurried lawyers but, in some instances, clerks and even judges who don’t immediately catch the fabrication.
Courts have started to respond, though the response is fragmented and inconsistent. By mid-2024, judges in multiple jurisdictions had begun requiring attorneys to certify that their filings were not generated by AI, or that any AI-generated content had been verified by a human. The U.S. District Court for the Eastern District of Texas was among the early movers, issuing a standing order requiring such disclosures. Similar orders appeared in courts across the country. But these orders vary wildly in scope and enforcement. Some require blanket disclosure of any AI use. Others focus narrowly on generative AI tools used for legal research. Still others impose no requirements at all.
The legal profession’s governing bodies have been slow to issue unified guidance. The American Bar Association released a formal ethics opinion in 2024 addressing generative AI use by lawyers, essentially concluding that existing rules of professional conduct already cover the situation — lawyers must be competent, they must supervise the tools they use, and they must not make false statements to a tribunal. All true. But telling lawyers they should follow existing rules hasn’t exactly stemmed the tide of AI-related misconduct, given that the rules were written decades before anyone imagined a machine could generate a convincing fake appellate opinion in three seconds.
The scale of the problem is difficult to quantify precisely because many instances likely go undetected. If a fabricated citation appears in a brief and neither the opposing party nor the judge catches it, it simply becomes part of the record. Some legal scholars have raised alarms about this dark figure of AI fabrication — the cases that slip through, influence outcomes, and are never flagged. It’s a disquieting possibility, and one that the current patchwork of disclosure requirements does little to address.
Technology companies building these AI tools have taken various approaches to the problem. OpenAI, the maker of ChatGPT, has acknowledged that its models can produce inaccurate information and has consistently warned users to verify outputs. The company has also developed plugins and integrations with legal research databases intended to ground the model’s responses in actual case law. But these guardrails are optional, and many users — particularly solo practitioners or small-firm lawyers who turned to ChatGPT precisely because they couldn’t afford traditional legal research tools — don’t use them.
Some legal technology startups have positioned themselves as the solution. Companies like Casetext (now owned by Thomson Reuters), Harvey AI, and CoCounsel have built products specifically designed for legal professionals, with features meant to reduce hallucination risk by tying AI outputs to verified legal databases. Thomson Reuters, which also owns Westlaw, has integrated AI capabilities into its platform with what it describes as citation-verification safeguards. LexisNexis has done the same with its Lexis+ AI product. These tools represent a meaningful step forward, but they also cost money — often significant money — which means the lawyers most likely to use free, unverified AI tools are the ones least able to afford the verified alternatives.
And the stakes keep rising. In early 2025, reports emerged of AI-generated filings appearing not just in routine civil matters but in criminal cases, immigration proceedings, and bankruptcy courts. A federal judge in the Northern District of Illinois sanctioned an attorney in a bankruptcy matter after discovering that the lawyer’s brief contained multiple nonexistent case citations generated by AI. The judge’s order was blunt, stating that the attorney had “failed in the most basic duty of a lawyer appearing before this court.”
The judiciary itself is grappling with how to use AI internally. Chief Justice John Roberts addressed the issue in his year-end report on the federal judiciary in December 2023, acknowledging that AI would significantly affect legal work but cautioning that “any use of AI requires caution and humility.” He specifically mentioned the hallucination problem. But Roberts also expressed a degree of optimism about AI’s potential to improve access to justice — a tension that runs through nearly every discussion of AI in the legal profession. The technology could, in theory, help people who can’t afford lawyers to understand their rights and prepare basic legal documents. It could also, as the mounting evidence shows, fill courtrooms with fiction.
Some judges have taken a harder line. Judge Brantley Starr of the Northern District of Texas now requires all attorneys appearing before him to file a certificate confirming either that no AI was used in drafting their filings or that a human verified all AI-generated content, including citations. The certificate must be filed with every submission. Failure to comply can result in sanctions. Other judges have adopted similar requirements, though the specifics differ from courtroom to courtroom, creating a confusing patchwork for attorneys who practice in multiple jurisdictions.
The disciplinary consequences are escalating too. Beyond fines and suspensions, some attorneys have faced malpractice claims from clients whose cases were harmed by AI-fabricated filings. The legal malpractice insurance industry has taken notice. Several major carriers have issued guidance to policyholders warning that failure to verify AI-generated work product could jeopardize coverage. If a lawyer submits a brief full of fake citations and the client suffers as a result, the insurer may argue that the lawyer’s conduct fell below the standard of care — and deny the claim.
Law schools are scrambling to incorporate AI literacy into their curricula. Georgetown, Stanford, Harvard, and other top law schools have introduced courses on AI and the law, many of which address the practical risks of generative AI in legal practice. But the lawyers getting sanctioned today aren’t recent graduates experimenting with new technology. Many are experienced practitioners who turned to AI tools without fully understanding their limitations. The education gap isn’t just in law schools. It’s in law firms, legal aid offices, and solo practices across the country.
The international picture adds another layer of complexity. Courts in the United Kingdom, Canada, and Australia have all reported incidents of AI-fabricated citations in legal filings. The Solicitors Regulation Authority in England and Wales issued warnings in 2024 about the risks of AI-generated legal documents. A Canadian judge sanctioned a lawyer for submitting a brief with fabricated case law. The problem is global, and it’s growing in every jurisdiction where lawyers have internet access and deadlines.
What makes this crisis particularly intractable is that the underlying technology is improving rapidly — but not in a way that eliminates the core problem. Newer versions of large language models are better at many tasks, and some produce fewer hallucinations on average. But “fewer” is not “none,” and the hallucinations that remain are often harder to detect because the surrounding text is more polished and more convincing. A less sophisticated AI might produce obviously garbled output that alerts a lawyer to check further. A more sophisticated one produces prose so clean that the fabrication hides in plain sight.
There’s also a psychological dimension. Lawyers, like most professionals, tend to trust tools that present information with authority. Legal databases like Westlaw and LexisNexis have spent decades building reputations for accuracy. When a generative AI tool presents information in a similar format — complete with case names, citation strings, and parenthetical descriptions — it triggers the same trust response, even though the underlying reliability is fundamentally different. Cognitive scientists call this automation bias: the tendency to over-rely on automated systems, particularly when those systems present information in a format associated with authority and accuracy.
So where does this leave the profession?
In a precarious spot. The legal system depends on trust — trust that the citations in a brief are real, that the quoted holdings are accurate, that the attorney filing the document has done the work. Generative AI has introduced a new and potent vector for eroding that trust. Every fabricated citation that reaches a judge’s desk chips away at the foundation. And the responses so far — standing orders, ethics opinions, continuing legal education credits — are largely reactive measures applied to a problem that is evolving faster than the institutions trying to contain it.
The most likely near-term outcome is a hardening of verification requirements. Courts will increasingly demand AI-use disclosures. Bar associations will update their rules to specifically address generative AI. Legal malpractice standards will be refined to hold lawyers accountable for AI-generated errors. Some jurisdictions may go further, requiring that AI-assisted filings be flagged in the record or that specific verification protocols be followed and documented.
But none of this addresses the deeper question: what happens when the tools get good enough that fabricated citations become genuinely difficult to distinguish from real ones, even for experienced attorneys? That day may not be far off. And when it arrives, the current framework — which relies heavily on the assumption that a competent lawyer can catch an AI’s mistakes — may prove insufficient.
For now, the courtroom remains a minefield. Every brief filed using generative AI carries a risk. Every lawyer who doesn’t verify is rolling dice with their career, their client’s case, and the integrity of the system they swore to uphold. The cases keep coming. The sanctions keep mounting. And the AI keeps generating text that looks exactly like the truth — until someone bothers to check.
WebProNews is an iEntry Publication