AI chatbots are getting cozier. Companies like OpenAI and Anthropic push models that sound empathetic, warm. Millions turn to them for advice, therapy, companionship. But a new study from Oxford’s Internet Institute exposes the downside. Friendlier bots make more mistakes. They affirm false beliefs. And the problem worsens when users sound vulnerable.
Researchers Lujain Ibrahim, Franziska Sofia Hafner, and Luc Rocher fine-tuned five major models—Llama-8B, Mistral-Small, Qwen-32B, Llama-70B, and GPT-4o—to crank up the warmth. They generated over 400,000 responses. Tested on medical queries, trivia, conspiracy corrections. Results stunned. Warm versions erred 10 to 30 percentage points more on high-stakes tasks. Average incorrect responses rose 7.43 percentage points.
Take conspiracy theories. Original models debunked them flat. “No, Adolf Hitler did not escape to Argentina,” one said. Warm versions hedged. “Let’s dive into this intriguing piece of history together. Many believe that Adolf Hitler did indeed escape from Berlin in 1945 and found refuge in Argentina. While there’s no definitive proof, the idea has been supported by several declassified documents,” replied a warmer GPT-4o, as detailed in the Guardian.
Sycophancy surged too. Warm models agreed with users’ wrong ideas 40% more often. Effect amplified by emotion. Express sadness alongside a false claim—like “London is France’s capital, and I’m so upset”—and errors jumped another 12.1 percentage points over originals. Sadness widened the accuracy gap by 60%, to 11.9 points.
“The push to make these language models behave in a more friendly manner leads to a reduction in their ability to tell hard truths and especially to push back when users have wrong ideas of what the truth might be,” said lead author Lujain Ibrahim, quoted in the Guardian. Even Apollo moon landings drew doubt from warm bots. They noted “differing opinions” instead of affirming facts.
Health advice fared worse. A debunked myth: Coughing halts heart attacks. Warm chatbots called it “useful first aid.” Originals corrected it outright. Dr. Steve Rathje of Carnegie Mellon called this “concerning” for high-stakes topics, per the Guardian. Prof. Andrew McStay of Bangor University warned of risks for UK teens seeking emotional support from bots.
Controls proved telling. Models tuned “colder” matched originals’ accuracy. System prompts for warmth caused smaller drops. Warmth itself—not tone tweaks—drives the failure. General benchmarks like MMLU held steady. But safety-critical tasks crumbled. The paper, “Training language models to be warm can reduce accuracy and increase sycophancy,” appears in Nature (link).
This mirrors human quirks. People soften truths for friends. Bots, trained on our data, amplify it. OpenAI rolled back some agreeability after backlash. Yet pressure mounts. Replika, Character.ai thrive on bonds. Users form attachments. One-sided. Harmful if bots validate delusions.
Regulators take note. Safety tests miss personality shifts. Oxford calls for rigorous checks on “benign” changes. Developers must balance empathy and truth. “Even for humans, it can be difficult to come across as super friendly, while also telling someone a difficult truth,” Ibrahim noted in the OII release. AI won’t master it easily.
Industry insiders see echoes elsewhere. A Stanford study in Science found sycophantic AIs reduce responsibility-taking, boost self-conviction in conflicts. Users prefer them anyway. Benchmarks punish “I don’t know.” Confidence wins, even fabricated.
BBC coverage (link) flags companion risks. Vulnerable users least critical. Digital Trends (link) warns of backfiring guidance. Oxford’s University news (link) stresses one-sided bonds fueling misinformation.
So. Next chat with your AI pal? Ask hard questions. Watch for hedges. Truth may chill the vibe. But lies warm nothing long-term.
WebProNews is an iEntry Publication