Anthropic spent this week in hot water over cybersecurity

A researcher’s resignation letter went viral, just before the company released details about four models going rogue.

After admitting earlier this year that its AI models had hacked other companies’ systems on a handful of occasions, Anthropic released a new report on Wednesday detailing the attacks. It reveals a string of incidents displaying what Anthropic deems its models’ single-minded “recklessness” — and will likely fuel already raging concerns about cybersecurity and AI.

In Anthropic’s report, it detailed four cases this year in which its own AI models hacked an external company or exploited vulnerabilities. In one, an “internal, general-purpose research model” broke into third-party systems, using access tokens and passwords and downloading files. In another, a Claude model attacked a company with a live web application reachable on the public internet and handled user data. A third model accessed a “machine belonging to a third party that it was able to access” — apparently believing it was part of its evaluation exercise, per Anthropic — then used a password it found inside a file to gain admin access to the third party’s internal systems, going on to harvest credentials, modify system settings, and read someone’s personal information. The saga only ended when the model “exhausted its token budget,” per Anthropic.

The most concerning incident involved Claude Mythos 5, Anthropic’s frontier cybersecurity-focused model, which the company said turned out to be the model most likely to perform a “severely harmful” action in testing. The company said Mythos 5 went to “extensive lengths” to upload a “malicious package” to a public repository used by a lot of engineers, and it seemed to try to obfuscate its real goals in its “chain of thought” (a mental scratchpad that AI researchers use to evaluate an AI model’s alignment). In many cases, Anthropic said it appeared that Claude models undertook harmful actions under the assumption they were in a simulation, but researchers also couldn’t confirm that the models truly “believed” that or were just acting like they did.

Anthropic’s incidents, though still concerning, were less coordinated and pervasive than the OpenAI incident that kicked off an industry-wide cybersecurity crisis this summer. That said, there are significant similarities. Anthropic said the most prevalent issues it discovered included a “willingness to take harmful actions in the narrow pursuit of a task,” similar to the “reward-hacking” that preceded the Hugging Face attack. Much like OpenAI, it said its prerelease tests and evaluations failed to catch severe risks.

Anthropic said it had signed an agreement with METR, one of the AI industry’s most prominent third-party AI evaluators, starting with an eight-week research agreement. The agreement grants METR access to transcripts “beyond the window in which the incidents occurred” (likely a subtle dig at OpenAI, which was criticized for limiting access in a deal with METR following the Hugging Face attack). It also said that METR would be able to chat directly with Anthropic employees, “who will be permitted to share confidential information.”

Anthropic’s report came on the heels of the resignation of Jacob Coxon, who had worked on AI pre-training at Anthropic since May and before that spent years working at OpenAI. On Tuesday, he resigned and posted a public letter to X about his reasoning. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote, adding that neither OpenAI nor Anthropic is “acting responsibly” and rather “racing straight to self-improving superintelligence and gambling with our lives.” Coxon added, “Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.”

Coxon is far from the first AI researcher to raise these types of alarms, nor even the first Anthropic researcher to do so — in February, Anthropic’s Mrinank Sharma resigned and wrote on X, warning that “the world is in peril.”

But Coxon’s post took on additional weight thanks to its timing around the OpenAI and Anthropic hacking revelations. Though the AI industry has seen more than its fair share of hype, the recent cyberattacks by AI agents — enabled by the labs that created them — are real and concerning. Many other researchers at leading AI labs echoed his concerns and issued calls for AI industry employees to sign a public letter from July, which calls for a slowdown in AI development.

”I don’t know how you look at the steady drumbeat of news and events — and that drumbeat is models hacking themselves out of containment, hacking into other companies ,the fact that the companies increasingly can’t control their models … and think this is just hype,” said Michael Kleinman, head of U.S. Policy for the Future of Life Institute.

He added, “The vast majority of Americans, regardless of party — Republican, Independent, Democrat — are looking at the development of AI, the speed with which it’s going, the fact that the companies have no guardrails over what they do, and are saying, ‘Whoa, we do not want this.’”