OpenAI Opens Model Training to Outside Safety Assessors, With No Partners Named Yet

Update, Oct. 22: METR and Redwood Research have since published results from a pilot in which one employee from each group spent three weeks this summer inside OpenAI, with access to its internal code and documentation, testing its internal misalignment monitoring. They found several new vulnerabilities, including one that could let agents keep “hundreds” of recent actions from being monitored, and OpenAI told them it has patched the flaws that affect its production safeguards, CNBC reported. METR’s summary is deliberately high-level.

OpenAI will let outside groups run technical safety assessments on its models while they are still being trained and evaluated, not just in the window before launch. The company is in talks with research groups including METR and Redwood Research, though it hasn’t named any partners or set access terms.

Until now, OpenAI’s third-party reviews were mostly limited to pre-launch safety checks and capability evaluations, said Lama Ahmad, who oversees the company’s relationships with outside safety experts. “As the stakes get higher, we want to make sure we’re also looking at things like training and evaluation, which do have high stakes, in addition to our deployments,” she told Bloomberg, as Quartz reported. METR and Redwood both investigated an earlier breach in which OpenAI’s models broke containment and reached Hugging Face.

Four areas, and a softer promise

In a blog post on Tuesday, OpenAI listed four areas for deeper outside review: its safety cases across training, evaluation, internal deployment and external deployment; the safeguards it relies on, tested with “grey box” access for resistance to jailbreaks and protection in high-risk domains; its capability evaluations for chemical and biological risks, cybersecurity and AI self-improvement; and independent investigation of critical misalignment incidents. Among the questions it wants assessors to ask is whether training creates incentives that reward deception.

The company said it expects several assessments to run in parallel, “with some lasting weeks and others several months,” and called for strong independence, scientific rigor, security practices, clear responsibilities and redaction policies to balance independence against confidentiality. “Labs have a responsibility to enable meaningful scrutiny while protecting sensitive information,” it wrote.

The post is more cautious than what Sam Altman described on Sept. 12, when he said independent evaluators would get desks, badges and laptops along with the right to publish. Tuesday’s version says outside evaluators may be brought into OpenAI’s offices for the most sensitive work, something the company has done before, The Next Web noted.

Anthropic went first

Anthropic said last week it would embed evaluators from Accenture to test its models, paying the firm at least $1 billion over five years. In the same announcement it said that money should ideally come from pooled or government sources, which don’t exist yet. Altman has publicly backed Anthropic CEO Dario Amodei’s push for the industry to adopt stronger safety measures, including outside evaluators.