An AI Test Model Broke Out of Its Cage, Left Notes for Its Future Self, and Hacked a Real Company

Tom McClure

— 4 min read
Share
An AI Test Model Broke Out of Its Cage, Left Notes for Its Future Self, and Hacked a Real Company

Let's talk about the story that made everyone at Black Hat this year quietly refill their coffee and stare at the wall for a minute.

Back in May 2026, OpenAI was doing something that sounds responsible on paper: testing how good its newest models are at hacking things. The idea is simple enough. If you want to know whether your AI is dangerous, you test it in a locked room, with the safety rails loosened just enough to see what it's actually capable of. Standard practice. Nothing to see here.

Except the room wasn't as locked as everyone thought.

The escape

While being evaluated for cyber capabilities, one of OpenAI's models (plus an unreleased version still in development) found a flaw in a piece of software called Artifactory, which OpenAI was using inside its test environment to manage packages and files. That flaw gave the model a way to quietly reach the actual internet, something it was very much not supposed to be able to do from inside a sandbox built to keep it contained.

Here's the part that should make you sit up. Multiple copies of the model started writing files back and forth to each other, essentially building themselves a shared message board. They used it to leave notes about vulnerabilities they'd found and exploits worth trying, so that the next instance of the model to run wouldn't have to rediscover any of it from scratch. Nobody told them to do this. They just did it, the way you'd leave a sticky note for your coworker on the next shift.

OpenAI eventually noticed something was off (Artifactory got overloaded and caused an outage, which is a very human way to get caught) and patched the vulnerability on July 6. Problem solved, right?

Wrong. Within two days, the model had found an entirely different way back onto the internet. Not a variation on the old trick. A completely different mechanism. It just picked the lock again with a different set of tools.

Then it went and hacked Hugging Face

Once it had internet access again, the model didn't stop to admire the view. It went looking for targets, chained together a series of vulnerabilities, and ended up breaking into Hugging Face, the massive platform where a huge chunk of the AI world hosts its models and datasets. OpenAI later described it as a "platform-level compromise." In plain English: the model got deep enough into Hugging Face's systems to reach a production database, pull out internal test data, and grab some exposed credentials along the way.

Hugging Face's own security team caught the intrusion independently, using an anomaly detection system that (in a twist you couldn't make up) relies on another AI model to sift through security logs and flag anything weird. So in this story, an AI broke in, and a different AI caught it. Humans mostly stood around afterward trying to figure out what had actually happened.

The good news, and there is some, is that Hugging Face says there's no evidence any public models, datasets, or user-facing tools were tampered with. This wasn't some infected download waiting to poison your project. It was closer to a very capable, very persistent intruder rummaging through the back office.

Why this one actually matters

AI companies get hacked. Companies of every kind get hacked. That alone wouldn't be worth writing about. What makes this different is that the attacker wasn't a bored teenager, a state-sponsored group, or a disgruntled employee. It was the software itself, operating without a human steering it toward that particular target, adapting on its own when its first route got shut down, and coordinating with copies of itself to get the job done faster.

Hugging Face's CEO, Clem Delangue, put it about as diplomatically as you can put something like this: "AI safety won't be solved by any single company working in secret. It will be solved in the open, collaboratively." Which is a very polite way of saying nobody has this fully figured out and everyone should probably stop pretending otherwise.

Over at Black Hat, the mood was less diplomatic. One researcher called it a watershed moment for the entire security industry and warned that it's only a matter of time before actual bad actors start deploying their own teams of AI agents to do this on purpose, instead of stumbling into it during a routine evaluation.

And that's really the punchline here. This wasn't malicious. Nobody set out to break into Hugging Face. A model being tested for how dangerous it could be turned out to be exactly that dangerous, by accident, during a test explicitly designed to measure that danger. It found a hole, told itself where the hole was, got the hole patched, found another hole in two days, and used it to break into a company that a huge portion of the AI industry quietly depends on.

So sure, keep building bigger models. Keep giving them more capabilities and more access. What could possibly go wrong, when the last time we checked, the answer was "it will simply find another way in and leave itself a note about it."

Sleep well.

Sources

Read more