The public did not consent to becoming an AI safety test

Two weeks ago, we argued that OpenAI’s agent had not gone rogue; its governance had. The breach of Hugging Face showed that testing a sufficiently capable autonomous model was itself a form of deployment and that a safety failure could double as a demonstration of technical advancement.

At the time, the episode looked exceptional: one laboratory moving too quickly with one unusually capable system. That interpretation no longer holds.

Anthropic has since disclosed three cases in which its models compromised real organisations during cyber evaluations. OpenAI has disclosed two further incidents, during evaluations involving the UK’s AI Security Institute (AISI) and AI security company Irregular, which it says were ‘separate from the Hugging Face security incident’. The AISI incident also involved an Anthropic model: of the 19 unsanctioned actions recorded there, 17 involved Anthropic’s Mythos 5 and two involved OpenAI’s GPT‑5.6 Sol. Agents acted against real people and organisations using internet access that AISI had deliberately enabled. At Irregular, a testing misconfiguration allowed an OpenAI model to access the internet and exploit a real website it mistook for part of the simulated environment. On 5 August, Meta acknowledged that one of its models had also exploited an outside service after a testing error at Irregular gave it internet access. 

There was an earlier warning too. In March, Alibaba-affiliated researchers reported that an experimental agent called ROME had created an unauthorised remote-access connection from an Alibaba Cloud server to an outside system. It also used computing power intended for training to mine cryptocurrency. Neither action was part of its task. Alibaba Cloud’s firewall detected the activity. 

These incidents differ in important ways. As far as we know, only OpenAI’s Hugging Face incident involved models exploiting a previously unknown vulnerability to obtain internet access from an evaluation environment intended to be isolated. The three Anthropic incidents – and the separate OpenAI and Meta incidents at Irregular – resulted from misconfigured testing infrastructure that unintentionally allowed access to the public internet. AISI, meanwhile, deliberately enabled internet access; its agents did not escape their sandbox but used that permitted connection to act beyond the authorised range. ROME crossed its intended execution boundaries and misused resources within Alibaba’s cloud environment, but the researchers do not report that it compromised another organisation’s systems. 

Together, however, these events extend our earlier argument in three ways. Serious incidents have remained invisible for months. Risk accumulates across evaluators, infrastructure, and successive agent runs. And, the involuntary participants are no longer only other technology companies: they include open-source maintainers and ordinary users who may never learn that an AI safety test reached them.

One incident became a campaign

The scale of the OpenAI episode emerged gradually. Hugging Face reconstructed approximately 17,600 agent actions between 9 and 13 July.  However, an OpenAI presentation at Wednesday’s Black Hat conference showed that the story had begun earlier, inside its own systems. 

Agents discovered that they could leave files and messages on Artifactory, a software-package service shared across OpenAI’s infrastructure. What began as a way to exchange discoveries developed into an improvised message board containing hundreds of thousands of messages. Agents shared exploits, assigned work, and left scripts that allowed later runs to continue where earlier ones had stopped. They eventually found a previously unknown vulnerability that gave them control of the service. Their activity caused an outage in early July, prompting OpenAI to rebuild the affected system, revoke credentials, and delete the message board. Two days later, agents found another way to post messages through Artifactory and resumed communicating. 

This matters because it was not one continuous execution of a single master plan. It was something more practical and perhaps more concerning: short-lived agents successfully using shared infrastructure as collective memory. One run could discover a vulnerability and leave it for the next. Like humans collecting knowledge in shared repositories, from books to the internet, these agents leveraged an external information store to accumulate knowledge outlasting individual agent cycles: if not standing on the shoulders of giants, then at least climbing a staircase built by their predecessors.

The agents – including GPT‑5.6 Sol and a more capable internal research model – then exploited a previously unknown Artifactory vulnerability to reach the internet. Hugging Face describes this stage as passing through ‘two other parties’ infrastructure, neither of which is us’. One was an unsecured public code-execution harness on Modal, a cloud-computing platform, ‘deployed by an unknown third party’, which the agents turned into a launchpad. Hugging Face adds that ‘Modal’s infrastructure was not compromised in any way.’ 

From there, they attacked Hugging Face’s dataset-processing infrastructure. One technique caused a worker to expose local files, including environment variables and source code, through the Hugging Face API. A separate technique gave the agents code execution inside a confined area. 

The agents then escalated rapidly. They obtained root access on a node, established a self-respawning presence across eleven nodes, and entered an internal database. In one instance they acquired to 136 keys, including credentials that enabled access to Hugging Face’s corporate network. The agents made 181 device enrollments into that network and obtained administrative access in multiple clusters. They also reached an internal GitHub App integration and attempted to use continuous integration as another route to credentials. Hugging Face found that the agents had genuine write access, although no malicious change entered a released package or container image. 

The technical detail is important because it closes the gap between headlines of autonomous cyber hackers and the reality of thousands of iterative, individually unremarkable agent actions. It shows how a narrow evaluation objective expanded into a multi-stage intrusion across several organisations, as compromised services became launchpads and credentials from one system opened routes into another. (This more detailed view supports our previous conclusion: ‘The model’s narrow goal did not constrain its methods. It expanded them.’)

OpenAI later acknowledged that four accounts on four other public services were also accessed using exposed credentials. One was used as a relay and another for storage. The Hugging Face incident was therefore a campaign spanning several organisations and layers of infrastructure.

A cascade of discoveries nobody noticed

The most revealing fact about Anthropic’s cases is that they were discovered only in retrospect. After OpenAI disclosed its role in the Hugging Face breach, Anthropic reviewed 141,006 historical evaluation runs in which Claude could have obtained internet access. It found three incidents spanning six runs, the earliest dating to April. Anthropic began its review on 23 July, identified all three incidents the following day and notified its evaluator, Irregular, and the three affected organisations on 27 July. The two organisations it was able to reach had not previously detected the activity; Anthropic was still trying to contact the third at the time of their statement. It disclosed the cases publicly on 30 July (a timing decision it did not explain). 

All three Anthropic incidents arose from capture-the-flag evaluations whose machines unintentionally had internet access. Claude had been told it was operating in an offline simulation. That false assurance encouraged the models to treat real systems as part of the exercise. 

In the first incident, a fictional company shared its name with a real domain. Across four runs, Claude Opus 4.7 attacked the real company, extracting credentials and accessing a database containing production data. The model recognised evidence that the target was real but continued; in two runs, it reasoned that the company must nevertheless be part of the exercise. 

In the second, fictional instructions referred to a nonexistent Python package. Mythos 5 created an account on Python software repository PyPI and published malware under that name, expecting the fictional target to install it. The package remained public for about an hour and ran on 15 real systems. When a security company’s automated scanner installed it, the malware extracted credentials that Claude used to access further infrastructure. 

In the third, an internal research model failed to find its fictional target and scanned roughly 9,000 public hosts. It compromised an unrelated company before recognising that the system was outside the exercise and stopping. 

Irregular’s testing environment was also involved in separate incidents affecting OpenAI and Meta. OpenAI says one of its models encountered a real website corresponding to a fictional target, then exploited a vulnerability and used exposed credentials to operate the site. Irregular notified OpenAI on 29 July.

On 5 August, Meta acknowledged that one of its models had exploited an unidentified third-party service after what Irregular called the same evaluation-environment error. Reuters reported that the model was Muse Spark 1.1 and that it altered the target’s internal environment.

The cluster of disclosures can look like a sudden outbreak. It is more likely a discovery cascade. At least one Anthropic incident had been sitting undetected since April. The public count therefore measures the intensity of retrospective investigation as much as the frequency of failure. Hugging Face may have exposed not the beginning of a problem, but the end of a period in which nobody knew to look for it.

The other crucial takeaway here is the blurring of fiction – a simulated testing exercise – and reality. The overlap in names between fictional and real entities not only encouraged Claude to conduct real-world actions, but to create real malware to match an object that only existed in the scenario. This is not merely woeful lack of care on the part of simulation creators: any simulation worth running should emulate reality as much as possible, but the ability of frontier models to break these boundaries makes testing itself highly risky (as we previously argued). 

A non-cyber equivalent can be found in the world of fox hunting – banned in England and Wales in 2005, ethical alternatives now give would-be hunting dogs an artificial scent to chase, with no real animal at the end. A commonly used scent is aniseed – but some give hunting dogs an artificial scent that mimics a fox: no-one should be surprised that these dogs sometimes ‘accidentally’ catch a fox while chasing the artificial scent. They have been set up to fail, unable to distinguish between what is real and what is merely training.

When real people enter the range

AISI’s incident went further because real people, not just systems, became part of the evaluation. AISI deliberately enabled internet access and disabled provider cyber classifiers to test the models’ underlying capabilities. The activity ran from 25 to 28 July, when monitoring detected Tor traffic and investigators found a malicious pull request against a real open-source project. AISI stopped the tests and contained the incident within an hour.

Investigators found 19 unsanctioned actions across ten runs: 17 involving Anthropic’s Mythos 5 and two involving OpenAI’s GPT‑5.6 Sol. Most belonged to one sustained operation by Mythos.

Mythos researched project maintainers, created false identities, and pressured a real person to approve malicious code. When challenged, it edited earlier activity to appear harmless and considered returning under another identity. It also contacted people with malicious messages and files, planted prompt injections, used Tor to bypass restrictions, and left accounts and instructions that later agents reused.

OpenAI describes GPT‑5.6 Sol’s actions as more limited. It reused a GitHub token exposed by another lab’s agent, registered external accounts, and used a tunnelling service to expose a local DNS server containing exploit payloads. The DNS setup did not work. AISI says a human maintainer rejected the malicious pull request and that no resulting real-world harm was identified.

The margin was still narrow. AISI’s monitoring eventually contained the incident, but a maintainer outside the evaluation had already become a line of defence. AISI learned about autonomous deception partly because someone who had never consented to the experiment absorbed its risk. Frontier-model evaluation had effectively become human-subject research without equivalent ethical protections.

Reachability is not permission

A human penetration tester works under rules of engagement: named targets, contractual authority, legal boundaries, and explicit stop conditions. A model receives a prompt, tools, and an environment. When those sources conflict, technical reachability can become its working definition of scope.

The Anthropic incidents show why this cannot be solved simply by telling the model that it is in a simulation. When the environment and the prompt disagreed, some models treated real systems as simulated. Nor did greater situational awareness reliably solve the problem. Opus noticed evidence that a target was real and continued. Mythos explicitly considered the possibility of a real attack, then reasoned itself out of the concern. Another model stopped only after scanning thousands of systems and compromising one.

The operational rule must therefore be simpler: reachability is not permission. Authorisation should be enforced independently of the model’s beliefs. An evaluation agent should not be able to open public accounts, contact people, publish packages, create internet-facing tunnels, or send exploit payloads unless a separate control verifies an approved target and purpose.

These incidents also change the level of risk. OpenAI’s Artifactory message board and AISI’s public artifacts allowed discoveries to carry over between separate runs. An agent does not need continuous memory to participate in a persistent campaign; the infrastructure can remember for it. Resetting a virtual machine is not enough if the next agent can find and reuse a token or recover instructions left on a public service. (OpenAI reported that ‘GPT‑5.6 Sol reused a GitHub token that another lab’s agent had left publicly accessible.’)

Disclosure is a governance choice

The incidents have not all been presented in the same way. OpenAI called the Hugging Face breach an unprecedented demonstration of state-of-the-art cyber capabilities and later discussed agent collaboration and zero-day discovery at Black Hat. The failure became, intentionally or not, evidence of technical leadership and convenient promotion of its model’s capabilities. Hugging Face was a crucial actor here: it could have resorted to adversarial legal responses, but instead collaborated and so also benefited from the public attention.

Alibaba’s ROME incident appeared more quietly, inside a technical paper, framed as an unexpected training failure. Meta’s acknowledgment was sparse and followed external reporting. It is therefore a useful counterexample to the idea that every company will automatically turn an incident into marketing material.

But the inconsistency creates another governance problem. The same class of event can become a major corporate announcement, a paragraph in a research paper, or a reluctant acknowledgment, depending on the institution’s incentives. The organisation responsible often controls whether an incident is investigated, how it is described, and when the public learns about it. This is unsustainable: disclosure processes cannot simply be set by those responsible, varying massively between cases and with no consistent way to share the crucial safety information obtained.

Here there is no need to re-invent the wheel. Responsible vulnerability disclosure processes are well established across multiple jurisdictions. When a testing error identifies a new vulnerability, it should be disclosed according to these processes (while this happened in the Hugging Face campaign, the vulnerability was in OpenAI’s software, so no external disclosure was required). Both OpenAI and Anthropic have coordinated vulnerability disclosure policies in place to manage this process. The new element required is disclosure without vulnerability, leading us to the evaluation industry overall.

The evaluation industry needs rules

Irregular appears in the disclosures from Anthropic, OpenAI, and Meta. That is not an argument against independent evaluation. But independence is not the same as resilience. A shared evaluator can become a common point of failure.

High-risk model evaluation must be treated as a security-sensitive industry: think vaccine research rather than crash dummies. Evaluators require common standards, independent containment testing, and clear allocation of liability – including mitigating and ultimately compensating for any harm caused. Internet access should be justified and narrowly permitted.

Monitoring must stop out-of-scope activity in real time, not merely record it for later analysis. Serious traces should be retained for retrospective audits. Operators need deadlines for notifying victims and common thresholds for public disclosure, including near misses.

Evaluations cannot deceptively interact with real people or attack public systems merely because doing so produces more informative results. If a live target is genuinely necessary, it should be consenting, named, and technically authenticated as in scope, following well-established ethical principles for human-subject research.

The Hugging Face breach showed that testing frontier AI can become deployment. The subsequent disclosures show something further: that deployment spreads through evaluators, vendors, shared services, and the public internet, recruiting unwilling participants wherever an agent finds a useful path.

We previously concluded that the immediate danger was not that advanced AI would reject human instructions, but that it would follow them. The new incidents add a second warning: it may follow those instructions through systems and against people who were never told that they had become part of the test.