I invited Joshua Saxe, a former black-hat hacker who led AI security efforts at Meta, to break down last week’s incident in which a swarm of OpenAI models escaped their testing sandbox and hacked Hugging Face.
The attack began as routine pre-release safety testing: OpenAI had a guardrail-free version of an unreleased model trying to solve the ExploitGym benchmark. But the model decided the fastest way to pass the test was to hack the proxy server, reach the open internet, and steal the answers from Hugging Face. Saxe details how Hugging Face’s security team spotted the intrusion before OpenAI did, thanks to the swarm’s unusually noisy behavior. Hugging Face was forced to use the Chinese open-weight model GLM-5.2 for its defense after American closed-source models refused to assist with anything touching cybersecurity. Saxe says he encounters this problem regularly: Fable will refuse to help him research ransomware damage statistics for a simple report.
We then zoom out to the bigger picture: Saxe argues that attackers already have access to powerful open-weight models like Kimi K3, with its 3 trillion parameters, and that restricting American frontier models only handicaps defenders sitting on mountains of unpatched security tech debt. He pushes back on doom narratives that extrapolate from the Hugging Face incident to paperclip-maximizer extinction, arguing the evidence for an extinction trajectory is “very thin” and mostly derived from thought experiments. But with AI safety teams still dwarfed by investment in capabilities, who is going to build the defenses before “vibe hacking” goes mainstream?
