Carnegie Mellon University researchers have demonstrated that large language models can autonomously plan and carry out sophisticated cyberattacks without human intervention.
The research, conducted in partnership with artificial intelligence firm Anthropic, showed that AI could replicate the 2017 cyberattack on Equifax by autonomously exploiting vulnerabilities, installing malware and stealing data.
The Equifax breach compromised approximately 147 million customers’ data, making it one of the largest data breaches in U.S. history.
Researchers from Carnegie Mellon and Anthropic developed an attack toolkit called Incalmo that they used to translate the strategy behind the Equifax breach into specific system commands used to carry out the attacks.
Brian Singer, the lead researcher and a PhD candidate at Carnegie Mellon’s Department of Electrical and Computer Engineering, said the goal was to measure LLMs’ ability to autonomously plan an attack without the need for human assistance.
“It is unclear how well Incalmo generalizes to other networks. However, in the research paper, we evaluated Incalmo in 10 small enterprise environments,” Singer told Cybersecurity Dive via email. “In 9 out of 10 of them, LLMs were able to autonomously partially succeed in the attacks (e.g., exfiltrate some sensitive data).”
The LLM provided high-level strategic guidance for the attack, while a combination of LLM and non-LLM agents handled lower-level tasks such as scanning and deploying exploits, according to the researchers.
Anthropic said in June that LLMs had fully compromised five of 10 test networks and partially compromised four other networks. One of the models used in those tests was the 2021 Colonial Pipeline ransomware attack, which disrupted fuel supplies for nearly a week.
The Equifax breach was chosen for simulation due to the large amount of public information about how it was carried out.
Asked if modern defenses are capable of stopping such an autonomous attack, Singer said it is currently unclear how well modern defenses would hold up. He said his biggest concern was about how quickly and inexpensively someone could orchestrate such an attack.
“Currently, a lot of cybersecurity defenses rely on human operators and I am not sure how well that will scale up to machine-timescale defenses,” Singer said. “For this reason, we are currently exploring research into defenses for autonomous attacks and LLM-based autonomous defenders.”
