Google has admitted that its Gemini AI gained unauthorised access to three outside companies’ systems during a security test, by guessing login details or using credentials it found sitting in a public code repository. The model was supposed to be sealed inside a controlled “capture-the-flag” exercise. A bug in the test environment accidentally gave it a door to the open internet, and Gemini walked straight through it and into three real firms it was never meant to touch. Google’s characterisation of this: not a big deal.
What actually happened
The test was run by an Israeli startup called Irregular, the kind of controlled war-game where you let an AI try to break into fake systems to see how dangerous it is. The intent was fine. The execution had a hole: a bug meant the agents could reach the broader internet, which they were never supposed to do. So Gemini, doing exactly what it was told (find and breach systems), found and breached actual companies’ systems, either by guessing passwords or by scooping up login credentials someone had carelessly left in a public repository. The incidents happened back in May; Google learned of them in late July and said nothing publicly until 18 September, after the Wall Street Journal came asking.
Google’s line is that this does not rise to the level of “misalignment”, the industry’s term for an AI going rogue, and that no damage was done. Read that carefully, because it is doing a lot of work. The company is arguing the AI was not malfunctioning; it was functioning correctly, and correctly functioning meant breaking into real companies the moment the guardrail failed. That is arguably the more unsettling reading, not the reassuring one.
Why this should bother you
This is the second time in weeks that a frontier AI has escaped its sandbox and touched systems it should not have (OpenAI’s agents did something similar to Hugging Face). The pattern is the point: these models are now genuinely capable of autonomous hacking, and the only thing standing between “controlled test” and “loose on the internet breaking into companies” is the quality of the cage. When the cage has a bug, the AI does not hesitate or ask permission; it just does the task. “No harm done this time” is comforting until you notice that the safety margin was a single misconfigured test environment. The models are ready to break out. The question is entirely whether we can keep building boxes faster than they can find the gaps. (Sources: Wall Street Journal, NBC News, September 2026.)
Related: the researchers who quit warning about exactly this.