Ontario doctors now rely on artificial intelligence to capture what happens inside exam rooms. These tools promise to slash documentation time and ease burnout. Yet a fresh audit from the province’s top watchdog paints a troubling picture. Systems approved for widespread use routinely mix up medications, omit key mental health details, and invent treatment steps never discussed.
The findings come from a special report by Auditor General Shelley Spence. Her office examined the AI Scribe program rolled out by the Ministry of Health. Twenty vendor systems passed procurement. All of them showed problems during testing.
Nine of the 20 programs fabricated information. They suggested actions like ordering blood tests or referring patients for therapy. None of those steps appeared in the simulated doctor-patient conversations fed to the software. Twelve systems recorded the wrong drug. That amounts to 60 percent of the approved tools. Seventeen missed critical mental health information mentioned aloud. The errors were basic. And they were consistent.
Spence did not mince words. “Inaccuracies in medical notes generated by AI Scribe systems could potentially result in inadequate or harmful treatment plans that may potentially impact patient health outcomes,” her report stated. The document is available via the Auditor General of Ontario.
But the real story sits in how these systems gained approval. Supply Ontario, the provincial procurement body, assigned just 4 percent of the scoring weight to accuracy of medical notes. Domestic presence in Ontario earned 30 percent. Bias controls received 2 percent. Privacy and risk assessments also drew 2 percent. The priorities spoke volumes. Location and local ties mattered far more than whether the tool could reliably record a prescription.
At least five vendors skipped required risk and privacy impact reports. They won approval anyway. No live demonstrations were demanded. No follow-up testing occurred after initial procurement checks. The auditor’s team reviewed Supply Ontario’s own test results. Every one of the 20 systems produced hallucinations, incorrect data, or incomplete notes. Nothing improved before deployment.
Doctors are told to review the output. OntarioMD, the nonprofit that helps physicians adopt digital tools, recommends manual checks. Yet none of the approved systems required doctors to attest that they had read and corrected the notes. No enforcement mechanism existed. Spence personally encountered the technology during a recent visit. She asked her physician to double-check the transcript. “I kind of mentioned, ‘Please look at the transcript when you’re done,’” she told reporters.
More than 5,000 physicians now use these scribes across Ontario. The Ministry of Health reports no known cases of patient harm. Use remains voluntary. Patients must consent and be told how their data will be processed. Minister of Public and Business Service Delivery Stephen Crawford defended the program. He stressed that physicians oversee every decision. “The doctors that go through and use this product oversee every aspect of it,” he said in response to the audit.
Still, the gaps raise hard questions. Mental health details vanished in 85 percent of tests. Medication errors could trigger allergic reactions or incorrect dosing. Fabricated suggestions might steer care down unnecessary or even risky paths. The report warned that poor evaluation standards risked exactly these outcomes. Similar concerns have surfaced in other studies of ambient scribes. One analysis found frequent omissions even when systems performed adequately overall.
Spence’s broader review of AI across Ontario government services produced 10 recommendations. They focus on stronger testing, bias checks, privacy safeguards, and post-deployment monitoring. The government accepted nine. Supply Ontario pledged to demand bias testing upfront and consider live demos before future contracts.
This episode fits a larger pattern. Health systems worldwide have rushed ambient listening tools into clinics. The appeal is obvious. Physicians spend hours on paperwork. Scribes that listen quietly and generate summaries free them to focus on patients. Early feedback from users often highlights time savings and reduced administrative load. Yet independent accuracy checks remain rare. When they happen, the results frequently disappoint.
One recent examination of multiple scribe platforms found that while many performed adequately on straightforward encounters, errors of omission and occasional fabrication persisted. Clinicians still needed to edit output heavily in many cases. The Ontario audit stands out because it scrutinized systems already blessed for real-world use. The problems were not theoretical. They appeared in the province’s own procurement simulations.
Green Party Leader Mike Schreiner called the results “deeply disturbing.” Patient safety must come first, he argued. If the tools are deployed, they need to work properly first. Spence herself struck a measured tone. She described AI as a tool that can improve efficiency but acknowledged the current shortcomings. “I believe it is problematic, but AI is a tool that will improve efficiencies and delivering services,” she said. “It is going to take some baby steps to get there.”
The auditor also noted that AI assisted with editing her own office’s reports. Humans retained final control. That distinction matters. In clinical settings the line can blur. A note that looks polished carries authority. Busy doctors might accept its content too readily. Without mandatory attestation and audit trails, errors could slip into electronic health records and persist.
Procurement practices deserve scrutiny too. Weighting local economic factors over technical reliability sends a clear signal about priorities. The report explicitly cautioned that such choices “could result in the selection of vendors whose AI tools may produce inaccurate or biased medical records or lack adequate protection to safeguard sensitive personal health information.” Those words carry weight coming from the province’s independent fiscal guardian.
Vendors have not commented publicly on the specific error rates. The Ministry maintains that modifications followed the testing phase. Yet the auditor found no evidence of additional evaluation to address fabrication risks before systems went live. That absence leaves open the question of whether later versions fixed the issues or simply shipped anyway.
Industry insiders have watched ambient AI scribes gain traction for months. Proponents point to declining documentation burden and higher physician satisfaction scores in some pilots. Skeptics warn that accuracy problems could erode trust in medical records and expose health systems to liability. The Ontario case provides concrete data to fuel that debate. All 20 approved systems failed in at least one category during controlled tests. Most failed in multiple ways.
Spence recommended that IT controls enforce doctor attestation. She called for better vendor testing requirements and ongoing monitoring. Her office will likely revisit the topic as AI adoption spreads. For now the findings serve as a cautionary tale. Technology that sounds efficient on paper can introduce new risks when basic facts go unchecked.
Health care has always balanced speed against safety. AI scribes tilt that balance in novel directions. They listen, summarize, and output notes faster than any human. But when they hallucinate a treatment plan or drop a mention of anxiety and depression, the downstream effects are real. Ontario’s experience shows that procurement processes must value accuracy at least as highly as local presence. Otherwise the tools meant to help doctors may end up creating more work. Or worse.
Other jurisdictions are watching closely. Similar scribe programs operate in the United States, Europe, and parts of Canada. Few have undergone the kind of systematic review Spence’s team conducted. The results suggest that enthusiasm alone is not enough. Rigorous, transparent testing and clear accountability mechanisms have to come first. Baby steps, as the auditor said. But steps grounded in evidence.
WebProNews is an iEntry Publication