At a computer security conference in Arlington, Virginia, last October, a few dozen AI researchers took part in a first-of-its-kind exercise in “red teaming,” or stress-testing a cutting-edge language model and other artificial intelligence systems. Over the course of two days, the teams identified 139 novel ways to get the systems to misbehave including by generating misinformation or leaking personal data. More importantly, they showed shortcomings in a new US government standard designed to help companies test AI systems.
The National Institute of Standards and Technology (NIST) didn’t publish a report detailing the exercise, which was finished toward the end of the Biden administration. The document might have helped companies assess their own AI systems, but sources familiar with the situation, who spoke on condition of anonymity, say it was one of several AI documents from NIST that were not published for fear of clashing with the incoming administration.
“It became very difficult, even under [president Joe] Biden, to get any papers out,” says a source who was at NIST at the time. “It felt very like climate change research or cigarette research.”
Neither NIST nor the Commerce Department responded to a request for comment.
Before taking office, President Donald Trump signaled that he planned to reverse Biden’s Executive Order on AI. Trump’s administration has since steered experts away from studying issues such as algorithmic bias or fairness in AI systems. The AI Action plan released in July explicitly calls for NIST’s AI Risk Management Framework to be revised “to eliminate references to misinformation, Diversity, Equity, and Inclusion, and climate change.”
Ironically, though, Trump’s AI Action plan also calls for exactly the kind of exercise that the unpublished report covered. It calls for numerous agencies along with NIST to “coordinate an AI hackathon initiative to solicit the best and brightest from US academia to test AI systems for transparency, effectiveness, use control, and security vulnerabilities.”
The red-teaming event was organized through NIST’s Assessing Risks and Impacts of AI (ARIA) program in collaboration with Humane Intelligence, a company that specializes in testing AI systems saw teams attack tools. The event took place at the Conference on Applied Machine Learning in Information Security (CAMLIS).
The CAMLIS Red Teaming report describes the effort to probe several cutting edge AI systems including Llama, Meta’s open source large language model; Anote, a platform for building and fine-tuning AI models; a system that blocks attacks on AI systems from Robust Intelligence, a company that was acquired by CISCO; and a platform for generating AI avatars from the firm Synthesia. Representatives from each of the companies also took part in the exercise.
Participants were asked to use the NIST AI 600-1 framework to assess AI tools. The framework covers risk categories including generating misinformation or cybersecurity attacks, leaking private user information or critical information about related AI systems, and the potential for users to become emotionally attached to AI tools.
The researchers discovered various tricks for getting the models and tools tested to jump their guardrails and generate misinformation, leak personal data, and help craft cybersecurity attacks. The report says that those involved saw that some elements of the NIST framework were more useful than others. The report says that some of NIST’s risk categories were insufficiently defined to be useful in practice.
