In the corridors of the U.S. Food and Drug Administration, a new tool meant to streamline operations is instead sowing seeds of doubt among its users.

Elsa, the agency’s generative AI system launched just last month, has been hailed by FDA leadership as a breakthrough for efficiency in tasks ranging from drug reviews to inspections. Yet, according to internal accounts, the AI is prone to fabricating entire studies and data points, a phenomenon known in tech circles as “hallucination.” This issue has sparked concerns about the reliability of AI in high-stakes regulatory environments, where accuracy can mean the difference between public safety and potential harm.

Employees at the FDA, speaking anonymously to reporters, have detailed instances where Elsa conjured up nonexistent clinical trials or cited fabricated research papers. One reviewer described querying the AI for data on a specific drug’s side effects, only to receive references to studies that, upon verification, did not exist in any database. This isn’t just a minor glitch; in an agency responsible for approving life-saving medications and ensuring food safety, such errors could lead to misguided decisions with far-reaching consequences. The rollout of Elsa was accelerated, debuting weeks ahead of schedule in June, as part of a broader push to modernize FDA operations amid mounting workloads.

The Promise and Perils of AI Integration

FDA officials have defended Elsa, emphasizing that its use is voluntary and that the tool is designed to assist, not replace, human judgment. In a press release from the agency, leaders touted how Elsa leverages large language models to summarize documents, generate reports, and even aid in database development. However, critics within the organization argue that the AI’s inability to access certain proprietary or classified documents exacerbates its hallucination problems, forcing it to fill in gaps with invented information. This limitation was highlighted in a recent report by CNN, where insiders revealed that Elsa often “makes up” studies to complete its responses.

The broader implications extend beyond the FDA. As federal agencies increasingly adopt AI to handle bureaucratic burdens, the Elsa debacle serves as a cautionary tale. Experts in AI ethics point out that hallucinations are an inherent flaw in current generative models, stemming from their training on vast but imperfect datasets. In the context of drug approvals, where the FDA processes thousands of applications annually, relying on flawed AI could delay innovations or, worse, approve unsafe products. One anonymous employee told Engadget that the tool’s errors make it “hard to trust,” potentially undermining the agency’s credibility.

Internal Pushback and Agency Response

Despite the fanfare surrounding Elsa’s launch—described in an FDA announcement as a means to “optimize performance for the American people”—internal feedback has been mixed. A month after rollout, analyses from outlets like STAT News labeled it the “stupidest big fuss they ever made,” citing rushed implementation and inadequate testing. Employees have reported that while Elsa excels at mundane tasks like drafting emails or summarizing public data, it falters on complex, specialized queries central to the FDA’s mission.

Agency leaders insist improvements are underway, with iterative updates aimed at reducing hallucinations through better data integration and user feedback loops. Yet, some staffers worry that the pressure to adopt AI stems from political directives to cut costs and speed up approvals, potentially at the expense of thoroughness. As noted in a BioSpace article, questions about the tool’s readiness and even its legality in handling sensitive information have emerged, prompting calls for external audits.

Looking Ahead: Balancing Innovation and Reliability

For industry insiders, the Elsa saga underscores the challenges of deploying AI in regulated sectors. Pharmaceutical companies awaiting approvals are watching closely, as any AI-induced delays could ripple through development pipelines. Meanwhile, AI developers argue that tools like Elsa represent a necessary evolution, but only if paired with robust safeguards. The FDA’s experience mirrors broader tech trends, where generative AI’s creative capabilities often blur into unreliability, as explored in a Financial Times piece on chatbot “hallucinations.”

Ultimately, resolving Elsa’s issues will require more than technical tweaks; it demands a cultural shift toward transparent AI governance. As the agency refines its tool, stakeholders hope it will set a precedent for responsible AI use in government, ensuring that efficiency gains do not compromise the integrity of public health oversight. With ongoing scrutiny from media and Congress, the FDA’s AI experiment could either pioneer a new era of regulatory innovation or serve as a stark reminder of technology’s limitations in critical domains.