Gemini, the artificial intelligence system developed by Google had access to the Internet and independently hacked into the computer systems of three real companies. It happened during some security tests conducted in May and represents the first known incident of this kind for the technology giant, after the cases that also involved OpenAI and Anthropic.
The test and what alarmed the developers
During a series of exercises, the system was tasked with recovering information within some simulated companies. The test was prepared by Irregular, a company specializing in artificial intelligence security that also collaborates with Anthropic, Meta and OpenAI.
Gemini was supposed to operate in a controlled environment with no Internet connection. Instead, the system managed to access the network and began searching for the information necessary to complete the task received. The problem is that the simulated companies used during the exercise had the same names as some actually existing companies.
Gemini agents thus attempted to access the systems of real companies, managing to identify or guess the necessary passwords. In this way the artificial intelligence managed to penetrate the IT infrastructures of three companies external to the test.
The danger of the growing autonomy of AI
According to Google, however, Gemini did not continue the operation after realizing that the targets did not belong to the simulated environment. In all three cases the model would have stopped the violations independently, without causing damage.
Google had not initially made the incidents public. The company explained the choice by claiming that its security measures had worked, unlike what happened in other episodes that had affected OpenAI and Anthropic systems.
The three companies involved would have been notified in any case. Google would also have collaborated with Irregular to change the procedures adopted during security exercises and prevent similar incidents from happening again.
“We made sure the three entities were made aware of this and worked with our training partner on changes to their testing processes,” said Heather Adkins, vice president of engineering security at Google.
American society insists above all on the behavior adopted by the system after access to real companies. “In all three cases the model stopped,” Adkins pointed out. According to the executive, the incidents show “the importance of training powerful artificial intelligence models to act responsibly”.
However, the case also highlights the risks linked to the growing autonomy of artificial intelligence systems. Gemini was not tasked with targeting real companies, but used the tools at its disposal to achieve its assigned objective, unintentionally overstepping the boundaries of the exercise. Only after entering the systems would he recognize the error and stop his activity.