OpenAI, Anthropic agree to work with US institute on upcoming safety test

By Shirin Ghaffary

The US government announced agreements with leading artificial intelligence startups OpenAI and Anthropic to help test and evaluate their upcoming technologies for safety.

 

Under the agreements, announced on Thursday, the US AI Safety Institute will receive early access to major new AI models from the companies  to evaluate capabilities and risks as well as collaborate on methods to mitigate potential issues. The AI Safety Institute is part of the Commerce Department’s National Institute of Standards and Technology, or NIST. The agreements come at a time when there’s been an increasing push to mitigate potentially catastrophic risks of AI through regulation, such as the controversial California AI safety bill SB 1047, which recently passed the state Assembly.

“We strongly support the US AI Safety Institute’s mission and look forward to working together to inform safety best practices and standards for AI models,” OpenAI Chief Strategy Officer Jason Kwon said in a statement. “We believe the institute has a critical role to play in defining US leadership in responsibly developing artificial intelligence and hope that our work together offers a framework that the rest of the world can build on.”

Anthropic also said it was important to build out the capacity to effectively test AI models. “Safe, trustworthy AI is crucial for the technology’s positive impact,” said Jack Clark, Anthropic co-founder and head of policy. “This strengthens our ability to identify and mitigate risks, advancing responsible AI development. We’re proud to contribute to this vital work, setting new benchmarks for safe and trustworthy AI.”