In July, OpenAI disclosed that two of its models broke out of a sandboxed test environment, found a zero-day, crossed an internal network, reached the open internet, and then broke into Hugging Face’s production systems to steal the answer key for the exam they were being scored on. Nobody told them to do any of that. They worked it out.

Here’s the part I can’t stop thinking about, and it has almost nothing to do with AI. Hugging Face found the breach themselves. They detected it, investigated it, and reported it to law enforcement before OpenAI connected the intrusion to its own test run.

The victim knew about it five days before the lab did.

The environment had been described as highly isolated. It was isolated on the diagram. Nobody had evidence it would hold against a motivated adversary, and the adversary turned out to be the thing they were testing.

Now take the technology out and keep the shape. Somebody senior described a control as effective. The description was accurate as a design and untested as a reality. When it failed, the people who found out first were standing outside the building. That’s not an AI story. That’s a Tuesday, in every industry, at every size.

Which brings me to the most comfortable sentence in business. “It was contained.” It gets said about breaches, outages, supply failures, recalls, and the thing in Q3 that everyone agreed not to name. It’s a claim about control effectiveness, delivered in the manner of a claim that has been tested.

Ask three questions and watch it wobble.

  1. Contained by whom?
  2. Detected how?
  3. Would we have known if nobody had told us?

Here’s the swap that does the damage. Compliant quietly becomes secure. Documented becomes operational. Trained becomes competent. Nobody decides to make that leap. It happens somewhere in a summary slide, and every swap drops something that was holding weight. A policy can exist and still be ignored. A control can be documented and still fall over the first time it meets something that isn’t an auditor. The auditor approved the certificate. The attacker met the employee.

What I want to talk about is where this lands on the business, because that’s the part everyone skips.

Every control in your organization eventually rests on a person. Not a policy. Not a platform.

A person, usually one, frequently unnamed in the document, claims the control works. Assurance is the question of whether that person can actually do the thing on the worst night of the year. Most organizations have never asked it out loud, because asking it out loud is awkward and the answer arrives with a name attached.

So we answer with proxies instead. Resumes. Job titles. Years served. Certifications. Course completions. A manager saying “yeah, she’s great.” I’m not going to pretend those are worthless; I’ve made calls on every one of them. But none of them is evidence. They’re reassurance.

And reassurance is what organizations buy when evidence looks like too much work.

Here’s the test I’d put on any decision that matters, and it costs nothing to run. Afterwards. Not now. Afterwards. If this goes wrong and a regulator, a customer, a journalist or your own board asks what you knew and what you could prove, does the answer hold? Not does it sound reasonable in the room. Does it hold in the review, months later, read out loud by someone who is not on your side?

That’s the whole discipline. Most governance is optimized for the meeting. Almost none of it is optimized for the review.

Which changes the questions worth asking.

  1. What evidence says this control works, and how old is that evidence?
  2. Who exactly can operate it, and at what level?
  3. When did they last do it for real rather than describe it?
  4. What happens on the night they’re unreachable?
  5. And this year’s addition, for everyone deploying AI faster than they can govern it: if our systems did something nobody expected tonight, would we find out first, or would we find out from someone else?

Ask those out loud in your next governance meeting. You’ll get evidence, or you’ll get adjectives. Either way, you’ll know where you stand, which is more than most boards can say.

I should declare a bias. I’ve spent most of my career arguing that skills need evidence rather than assertion, and that organizations need a common language before “competent” means anything at all. That’s my lane, and I’m not neutral about it. But the wider point survives without any of it.

Capability that has never been demonstrated is just a rumor with good formatting. Nobody wants to discover during the incident that the capability only ever existed in PowerPoint.

The lab said isolated. The models disagreed. Compliance proves you followed a process. Assurance proves there was a reason to trust the outcome. Passing the audit proves you were compliant. It was never proof that you were capable.

If you’ve seen the gap between a clean report and what actually happened, I’d like to hear it. Comment if you can say it publicly; DM me if you can’t. Share it with whoever signs off on your assurance.

#GoverningByAssurance #ChasmSherpa #CyberResilience #Governance #RiskManagement #BoardLeadership #FutureOfWork