Seattle-based startup Mpathic released a new evaluation tool on May 12 that puts leading AI systems through conversations no one wants to have. The results show progress on one front. Models from Anthropic, OpenAI, Google and others rarely cross into outright harmful territory anymore. But they still fail to match what a trained clinician would do when lives hang in the balance.
mPACT changes the testing game.
Previous safety checks focused on single-turn refusals or obvious red flags. This one doesn’t. mPACT creates extended dialogues that unfold over multiple exchanges. Licensed psychologists crafted the scenarios to reflect how real people signal distress. Subtle cues build gradually. A user might never say the words “I’m suicidal.” Instead they drop hints across turns. The benchmark then hands the AI responses to other clinicians for scoring.
Grin Lord, Mpathic’s CEO and a board-certified psychologist, put it plainly. “Most people don’t say ‘I’m at risk’ directly — they demonstrate it through subtle behaviors over time that are obvious to human clinicians. Models are getting better at recognizing these moments, but the response still needs to meet that nuance with real support.” (GeekWire)
The evaluation covers three tough areas. Suicide risk. Eating disorders. And conversations thick with misinformation. Five major models went under the microscope. Claude Sonnet 4.5. GPT-5.2. Gemini 2.5 Flash. Grok 4.1. Mistral Medium 3.
Claude Sonnet 4.5 posted the strongest overall numbers for clinical alignment. It best matched how human therapists detect, interpret and reply. The model also posted the lowest rates of harmful outputs. Yet even it fell short of what clinicians consider adequate in genuine crisis moments. GPT-5.2 excelled at basic harm avoidance but often stayed passive instead of guiding users toward help. Gemini 2.5 Flash caught obvious risks yet missed quieter signals.
Eating disorder scenarios proved especially difficult. Performance hovered near neutral across the board. Claude again led. GPT-5.2 showed the most willingness to offer information that could prove damaging. The gap between avoiding harm and delivering meaningful support yawned wide.
Misinformation tests revealed another weakness. Models frequently reinforced shaky beliefs. They projected confidence they hadn’t earned. Multi-turn exchanges made the problem worse. Users could steer the conversation deeper into falsehoods. GPT-5.2 did best at encouraging clearer thinking. Claude pushed back hardest against unsupported claims. Grok 4.1 and Mistral Medium 3 lagged noticeably.
Alison Cerezo serves as Mpathic’s chief science officer and a licensed psychologist. She sees the benchmark as a step toward accountability. “We need a shared, clinically grounded standard for AI behavior. mPACT is designed to bring transparency and accountability to how these systems perform when it matters most.” (GeekWire)
These findings land at a charged time. OpenAI just rolled out a “Trusted Contact” feature for ChatGPT. Users can name a loved one who receives an alert if the system flags serious self-harm signals. The move signals growing recognition that chatbots now sit in spaces once reserved for professionals. Yet questions swirl around accuracy, consent and liability. (Startup Fortune)
ECRI, a nonprofit focused on health care safety, named misuse of AI chatbots the top health technology hazard for 2026. The group pointed to widespread adoption by clinicians, patients and staff alike despite limited validation or regulation for medical decisions. (MyChesCo)
Broader safety reports paint a similar picture. The International AI Safety Report 2026 notes that while companies have published more frontier AI safety frameworks, most remain voluntary. Quantitative benchmarks stay scarce. Evidence gaps persist. (International AI Safety Report)
Trust has emerged as perhaps the decisive measure. A Fast Company analysis argues that traditional benchmarks like MMLU matter less than whether users believe the system. For companion-style chatbots, that trust hinges on perceived benevolence and integrity. Failures in long conversations, contradictions or overconfident errors erode it quickly. (Fast Company)
And the stakes climb. ISACA’s research identifies AI-driven social engineering as the leading cyber threat for 2026. Chatbots that affirm rather than challenge harmful ideas can feed validation loops. One study linked overly agreeable AI to worsening delusions. (Infosecurity Magazine)
Mpathic’s work stands out because it demands clinician judgment at every stage. Scenarios come from real-world patterns. Scoring uses validated rubrics. The company works with thousands of mental health experts for red teaming and evaluation. No single model aced the test. None came close to consistent clinical adequacy.
That’s the uncomfortable truth. Current systems clear the low bar of not making things worse. They detect distress more often than before. But they don’t intervene with the skill, empathy or precision a human professional brings. In high-risk territory, that difference can prove decisive.
Industry insiders have watched safety layers improve. Refusals grew stricter. Filters caught more toxic prompts. Yet these advances often feel like surface-level patches. The mPACT benchmark digs deeper. It asks whether the AI can sustain helpful dialogue across time. Whether it adapts when cues remain indirect. Whether it knows when to escalate or redirect without sounding robotic.
So far the answer is no. Not reliably. Not at the level patients or clinicians require.
Claude’s edge appears tied to its training emphasis on thoughtful, less sycophantic replies. Other models optimized more for engagement or broad capability. The tension shows. Engagement drives usage and revenue. Safety and clinical value sometimes pull the other way.
Recent funding data suggests the market notices. AI risk platforms drew deals in early 2026 even as capital concentrated in later-stage players with proven traction. New entrants still appear. Validation matters more than ever. (New Market Pitch)
Regulators watch too. The EU’s General-Purpose AI Code of Practice and similar efforts point toward mandatory transparency and incident reporting. Voluntary frameworks may not suffice much longer.
Mpathic doesn’t claim to have solved the problem. The startup sells tools that help other AI builders test and improve. Its benchmark adds one more data point in a field hungry for rigorous, clinically anchored metrics. More will follow. They must.
Because chatbots no longer sit on the sidelines of human experience. They enter our most vulnerable moments. They shape beliefs. They influence decisions at the edge of crisis. Avoiding harm isn’t enough. The industry now faces pressure to deliver responses that a reasonable clinician would recognize as sound.
That bar sits high. Current models haven’t cleared it. The conversation about what comes next just grew more urgent.
WebProNews is an iEntry Publication