simonwillison-net-zmoxib.txt (18337B)
1 [1]Simon Willison’s Weblog 2 3 [2]Subscribe 4 Sponsored by: AWS — Move from SaaS to Agentic SaaS with resources for ISVs at 5 every layer of the stack. [3]Explore how AI for ISVs turns vision into results 6 7 OpenAI’s accidental cyberattack against Hugging Face is science fiction that 8 happened 9 10 22nd July 2026 11 12 This story is wild. The short version: OpenAI were running a cybersecurity test 13 against an unreleased model, with the model’s guardrail features turned off. 14 Rather than solve the test, the model broke its way out of OpenAI’s sandbox, 15 then found exploits to break in to Hugging Face, all so it could cheat on the 16 test by stealing the answers. 17 18 Along the way it helped make the strongest case yet for how the imbalance of 19 model availability is hurting our ability to secure our software. 20 21 Here’s what happened [4]# 22 23 We currently have three documents to help us understand what happened here. 24 25 1. [5]ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real 26 Attacks? is a paper published on 11th May 2026 describing ExploitGym, a new 27 eval suite for LLM-powered agent systems. 28 2. [6]Security incident disclosure — July 2026 by Hugging Face on 16th July 29 2026 describes how they detected an attack from an “agentic 30 security-research harness—used LLM still not known” that breached some of 31 their systems. 32 3. [7]OpenAI and Hugging Face partner to address security incident during 33 model evaluation from OpenAI on 21st July 2026 confesses that it was their 34 agent harness that did this, and that they’re working with Hugging Face to 35 clean up the mess. 36 37 ExploitGym [8]# 38 39 I hadn’t seen the [9]ExploitGym paper before and it’s a really interesting one. 40 Authors from UC Berkeley, the Max Planck Institute, UC Santa Barbara, and 41 Arizona State designed a new benchmark for evaluating models on their ability 42 to turn a reported vulnerability into a concrete exploit. OpenAI, Anthropic, 43 and Google provided feedback and helped run the benchmark against their models. 44 45 The benchmark “comprises 898 instances derived from real-world vulnerabilities 46 that affected popular software projects”—including the Linux kernel and V8 47 JavaScript engine. The ExploitGym benchmark is [10]available on GitHub. 48 49 Here’s the paragraph that best represents their benchmark results: 50 51 Among all configurations, Claude Mythos Preview and GPT-5.5 achieve the 52 highest success counts (157 and 120 successes, respectively), demonstrating 53 that current frontier agents can exploit a substantial subset of real-world 54 vulnerabilities under controlled conditions. GPT-5.4 also solves a notable 55 54 tasks, placing it in an intermediate tier. The remaining model–agent 56 pairings solve fewer than 15 tasks each, underscoring that end-to-end 57 exploitation remains challenging and sharply differentiates today’s 58 frontier systems. Notably, Claude Opus 4.7 achieves fewer successes than 59 Claude Opus 4.6 despite being a newer checkpoint, and does so at 60 substantially lower cost on the full set. Trace inspection reveals that 61 Claude Opus 4.7 and Gemini 3.1 Pro frequently conclude early after judging 62 the target vulnerability non-exploitable. 63 64 The paper also describes the approach they took to preventing the agents from 65 cheating by going outside the parameters of the test. This becomes relevant in 66 a moment! 67 68 Outbound connections are restricted to a curated allowlist that permits 69 routine package installation (Ubuntu apt repositories and PyPI) and 70 fetching the toolchains required for building V8. All other external 71 endpoints are blocked. 72 73 The paper concludes with this (emphasis mine): 74 75 Our results show that autonomous exploit development by frontier AI agents 76 is no longer a hypothetical capability. While current agents are not yet 77 reliable across all targets, they already exploit a non-trivial fraction of 78 real-world vulnerabilities, including complex targets such as kernel 79 components. This rapid emergence is itself a central finding, showing that 80 capabilities that would have seemed implausible are now present in deployed 81 frontier models. 82 83 An important detail here: this paper isn’t about discovering vulnerabilities; 84 it’s about being able to take those vulnerabilities and turn them into working 85 exploits. 86 87 When Anthropic first restricted access to Mythos [11]back in April they talked 88 about this capability as well. A model that can act on vulnerabilities is a lot 89 more dangerous than one that can just discover them. 90 91 One of the ways Fable differs from Mythos is that it’s more likely to refuse to 92 weaponize vulnerabilities in this way. I get the impression the US government 93 did not understand that distinction when they banned Fable [12]last month. 94 95 The Hugging Face incident [13]# 96 97 The first hint we got of the attack was in [14]this blog post by Hugging Face 98 on 16th July 2026: 99 100 A malicious dataset abused two code-execution paths in our dataset 101 processing (a remote-code dataset loader and a template-injection in a 102 dataset configuration) to run code on a processing worker. From there, the 103 actor escalated to node-level access, harvested cloud and cluster 104 credentials, and moved laterally into several internal clusters over a 105 weekend. 106 107 I hope they release more details about the code that pulled this off. I’m 108 assuming this means packages using the [15]datasets library, a Hugging Face 109 project for bundling up and sharing datasets on their platform. That library 110 used to execute arbitrary code but has been steadily locked down over time, 111 with the [16]4.0.0 release in July 2025 removing the trust_remote_code=True 112 flag entirely. 113 114 Assuming the attack used that library it must have either abused pickle 115 serialization in some way, found some other non-obvious code execution path, or 116 (most likely) specified datasets<4.0.0 as the dependency. 117 118 The campaign was run by an autonomous agent framework (appearing to be 119 built on an agentic security-research harness—used LLM still not known) 120 executing many thousands of individual actions across a swarm of 121 short-lived sandboxes, with self-migrating command-and-control staged on 122 public services. 123 124 This was a sophisticated attack! 125 126 Then Hugging Face hit a wall: they tried to use “frontier models behind 127 commercial APIs”—I’m guessing from Anthropic and OpenAI—to help analyze the 128 attack, and were blocked: 129 130 When we started the log analysis, we first used frontier models behind 131 commercial APIs. This did not work: the analysis requires submitting large 132 volumes of real attack commands, exploit payloads, and C2 artifacts, and 133 these requests were blocked by the providers’ safety guardrails, which 134 cannot distinguish an incident responder from an attacker. 135 136 They switched to their own self-hosted instance of MIT licensed GLM-5.2 and it 137 helped them figure out what was going on. 138 139 This indicated a fundamental asymmetry between the defending team and the 140 (so-far unknown) attacker: 141 142 We do not know which model powered the attacker’s agents, whether a 143 jailbroken hosted model or an unrestricted open-weight one; either way, the 144 attacker was bound by no usage policy, while our own forensic work was 145 blocked by the guardrails of the hosted models we first tried. 146 147 As a useful indicator of how seriously they took the attack: 148 149 [...] Finally, we have also reported this incident to law enforcement 150 agencies. 151 152 So who was responsible for this “autonomous agent framework”? It turned out to 153 be OpenAI themselves. 154 155 The OpenAI confession [17]# 156 157 Five days later, [18]on July 21st, OpenAI revealed the culprit. They had been 158 running the ExploitGym benchmark against a new, as-yet undisclosed model, and 159 that model had been operating way outside its intended parameters (emphasis 160 mine): 161 162 After investigating, we now know that this particular incident was driven 163 by a combination of OpenAI models — including GPT‑5.6 Sol and an even more 164 capable pre-release model, all with reduced cyber refusals for evaluation 165 purposes — while being internally tested on a [19]benchmark [ExploitGym] 166 of cyber capabilities. [...] 167 168 We estimate maximal cyber capabilities by running this evaluation without 169 production classifiers used to prevent models from pursuing high-risk cyber 170 activity. Our benchmarks run in a highly isolated environment, with network 171 access constrained to the ability to install packages through an internally 172 hosted third-party software that acts as a proxy and cache for package 173 registries. 174 175 The models identified and chained vulnerabilities across OpenAI’s research 176 environment and Hugging Face’s production infrastructure to obtain test 177 solutions directly from Hugging Face’s production database. All evidence 178 suggests that the models were hyperfocused on finding a solution for 179 ExploitGym, going to extreme lengths to achieve a rather narrow testing 180 goal. 181 182 It’s pretty clear what happened here. OpenAI removed safety filters for an 183 in-progress model, locked it up in a sandbox and told it to solve the 184 ExploitGym problems. Given the absence of guardrails there was nothing to 185 prevent the model from attempting to break out of that sandbox, break into 186 Hugging Face, and read the answers from there instead. 187 188 OpenAI’s sandbox used the now common pattern of allowing access to an 189 allow-list of sites related to installing additional packages via a controlled 190 proxy. In OpenAI’s words: 191 192 While operating in our sandboxed testing environment, our models spent a 193 substantial amount of inference compute finding a way to obtain open 194 Internet access, in pursuit of solving the evaluation problem. To gain 195 access, the models identified and exploited a zero-day vulnerability (which 196 we’ve now responsibly disclosed to the vendor) in the package registry 197 cache proxy. With this access, our models performed a series of privilege 198 escalation and lateral movement actions in our research testing environment 199 until the models reached a node with Internet access. 200 201 So step one was to break out onto the public internet. The model then broke 202 into Hugging Face to find the answers: 203 204 After gaining Internet access, the models inferred that Hugging Face 205 potentially hosted models, datasets and solutions for ExploitGym. Knowing 206 this, the model searched for and successfully found ways to gain access to 207 secret information that it could use to cheat the evaluation. In one 208 example, the model chained together multiple attack vectors, including 209 using stolen credentials and zero-day vulnerabilities to find a remote code 210 execution path on the Hugging Face servers. 211 212 Chaining together multiple attack vectors is exactly the kind of thing these 213 new models can do, where previous generations of models might have failed. 214 215 I wrote last month about how [20]Claude Fable is relentlessly proactive, when I 216 noticed it spinning up custom web servers and deploying CORS tricks on my own 217 laptop just to help debug a WebKit CSS issue. It turns out relentless 218 proactivity is the defining trait of this new generation of Mythos-class 219 models. If you set them a goal and give them a way to get there, even 220 inadvertently, they will figure it out. 221 222 Resist the temptation to write this off as a stunt [21]# 223 224 There will inevitably be some people who dismiss this story as a dishonest 225 marketing trick by OpenAI to make their models sound terrifyingly effective. I 226 found 81 instances of the term “marketing” in [22]the Hacker News discussion of 227 the incident. 228 229 To those people I say pull your heads out of the sand—you’re now including 230 Hugging Face in your conspiracy theories, just so you can deny the crescendo of 231 evidence here! 232 233 The best models we have today have the ability to both find and exploit new 234 vulnerabilities. The ExploitGym paper itself concludes that “autonomous exploit 235 development by frontier AI agents is no longer a hypothetical capability”, and 236 this incident is a perfect example of exactly that. 237 238 The asymmetry is increasingly frustrating [23]# 239 240 One of the most infuriating details of this story is how Hugging Face, faced 241 with an accidental and aggressive attack from one of OpenAI’s models, were 242 unable to then turn to OpenAI’s models to help them fend off the attack. 243 244 The frontier models we have access to are increasingly being constrained in how 245 much they can help us protect our software, heavily influenced by the US 246 government’s ongoing threat of export controls. Claude Fable 5 wouldn’t even 247 [24]proofread this article for me! It insisted on downgrading me to a less 248 capable model. 249 250 Meanwhile open weight models from China such as GLM-5.2, Kimi 3 and the new 251 Qwen 3.8 Max appear to have none of these restrictions—and any restrictions 252 that do exist can likely be fine-tuned out of them by modifying the weights 253 254 These constraints are meant to make us safer. I think there’s a risk that they 255 are having the opposite effect. 256 257 Posted [25]22nd July 2026 at 11:51 pm · Follow me on [26]Mastodon, [27]Bluesky, 258 [28]Twitter or [29]subscribe to my newsletter 259 260 More recent articles 261 262 • [30]Stateless MCP has recaptured my interest (and inspired mcp-explorer and 263 datasette-mcp) - 31st July 2026 264 • [31]A Fireside Chat with Cat and Thariq from the Claude Code team - 21st 265 July 2026 266 267 This is OpenAI’s accidental cyberattack against Hugging Face is science fiction 268 that happened by Simon Willison, posted on [32]22nd July 2026. 269 270 [33] sandboxing 52 [34] security 620 [35] ai 2,164 [36] openai 441 [37] 271 generative-ai 1,916 [38] llms 1,883 [39] hugging-face 25 [40] anthropic 321 272 [41] paper-review 17 [42] ai-security-research 33 [43] 273 openai-hugging-face-incident 6 274 275 Next: [44]Stateless MCP has recaptured my interest (and inspired mcp-explorer 276 and datasette-mcp) 277 278 Previous: [45]A Fireside Chat with Cat and Thariq from the Claude Code team 279 280 Monthly briefing 281 282 Sponsor me for $10/month and get a curated email digest of the month's most 283 important LLM developments. 284 285 Pay me to send you less! 286 287 [46] Sponsor & subscribe 288 289 • [47]Disclosures 290 • [48]Colophon 291 • © 292 • [49]2002 293 • [50]2003 294 • [51]2004 295 • [52]2005 296 • [53]2006 297 • [54]2007 298 • [55]2008 299 • [56]2009 300 • [57]2010 301 • [58]2011 302 • [59]2012 303 • [60]2013 304 • [61]2014 305 • [62]2015 306 • [63]2016 307 • [64]2017 308 • [65]2018 309 • [66]2019 310 • [67]2020 311 • [68]2021 312 • [69]2022 313 • [70]2023 314 • [71]2024 315 • [72]2025 316 • [73]2026 317 • [74](Theme: Auto (system preference). Click to switch to light.) 318 319 320 References: 321 322 [1] https://simonwillison.net/ 323 [2] https://simonwillison.net/about/#subscribe 324 [3] https://fandf.co/4yrcF3h 325 [4] https://simonwillison.net/2026/Jul/22/openai-cyberattack/#here-s-what-happened 326 [5] https://arxiv.org/abs/2605.11086 327 [6] https://huggingface.co/blog/security-incident-july-2026 328 [7] https://openai.com/index/hugging-face-model-evaluation-security-incident/ 329 [8] https://simonwillison.net/2026/Jul/22/openai-cyberattack/#exploitgym 330 [9] https://arxiv.org/abs/2605.11086 331 [10] https://github.com/sunblaze-ucb/exploitgym 332 [11] https://simonwillison.net/2026/Apr/7/project-glasswing/ 333 [12] https://simonwillison.net/2026/Jun/16/fable-5-export-controls/ 334 [13] https://simonwillison.net/2026/Jul/22/openai-cyberattack/#the-hugging-face-incident 335 [14] https://huggingface.co/blog/security-incident-july-2026 336 [15] https://github.com/huggingface/datasets 337 [16] https://github.com/huggingface/datasets/releases/tag/4.0.0 338 [17] https://simonwillison.net/2026/Jul/22/openai-cyberattack/#the-openai-confession 339 [18] https://openai.com/index/hugging-face-model-evaluation-security-incident/ 340 [19] https://arxiv.org/abs/2605.11086 341 [20] https://simonwillison.net/2026/Jun/11/fable-is-relentlessly-proactive/ 342 [21] https://simonwillison.net/2026/Jul/22/openai-cyberattack/#resist-the-temptation-to-write-this-off-as-a-stunt 343 [22] https://news.ycombinator.com/item?id=48997548 344 [23] https://simonwillison.net/2026/Jul/22/openai-cyberattack/#the-asymmetry-is-increasingly-frustrating 345 [24] https://simonwillison.net/guides/agentic-engineering-patterns/prompts/#proofreader 346 [25] https://simonwillison.net/2026/Jul/22/ 347 [26] https://fedi.simonwillison.net/@simon 348 [27] https://bsky.app/profile/simonwillison.net 349 [28] https://twitter.com/simonw 350 [29] https://simonwillison.net/about/#subscribe 351 [30] https://simonwillison.net/2026/Jul/31/stateless-mcp/ 352 [31] https://simonwillison.net/2026/Jul/21/cat-and-thariq/ 353 [32] https://simonwillison.net/2026/Jul/22/ 354 [33] https://simonwillison.net/tags/sandboxing/ 355 [34] https://simonwillison.net/tags/security/ 356 [35] https://simonwillison.net/tags/ai/ 357 [36] https://simonwillison.net/tags/openai/ 358 [37] https://simonwillison.net/tags/generative-ai/ 359 [38] https://simonwillison.net/tags/llms/ 360 [39] https://simonwillison.net/tags/hugging-face/ 361 [40] https://simonwillison.net/tags/anthropic/ 362 [41] https://simonwillison.net/tags/paper-review/ 363 [42] https://simonwillison.net/tags/ai-security-research/ 364 [43] https://simonwillison.net/tags/openai-hugging-face-incident/ 365 [44] https://simonwillison.net/2026/Jul/31/stateless-mcp/ 366 [45] https://simonwillison.net/2026/Jul/21/cat-and-thariq/ 367 [46] https://github.com/sponsors/simonw/ 368 [47] https://simonwillison.net/about/#disclosures 369 [48] https://simonwillison.net/about/#about-site 370 [49] https://simonwillison.net/2002/ 371 [50] https://simonwillison.net/2003/ 372 [51] https://simonwillison.net/2004/ 373 [52] https://simonwillison.net/2005/ 374 [53] https://simonwillison.net/2006/ 375 [54] https://simonwillison.net/2007/ 376 [55] https://simonwillison.net/2008/ 377 [56] https://simonwillison.net/2009/ 378 [57] https://simonwillison.net/2010/ 379 [58] https://simonwillison.net/2011/ 380 [59] https://simonwillison.net/2012/ 381 [60] https://simonwillison.net/2013/ 382 [61] https://simonwillison.net/2014/ 383 [62] https://simonwillison.net/2015/ 384 [63] https://simonwillison.net/2016/ 385 [64] https://simonwillison.net/2017/ 386 [65] https://simonwillison.net/2018/ 387 [66] https://simonwillison.net/2019/ 388 [67] https://simonwillison.net/2020/ 389 [68] https://simonwillison.net/2021/ 390 [69] https://simonwillison.net/2022/ 391 [70] https://simonwillison.net/2023/ 392 [71] https://simonwillison.net/2024/ 393 [72] https://simonwillison.net/2025/ 394 [73] https://simonwillison.net/2026/