While artificial intelligence advancements unlock opportunities in various industries, innovations may also become targets of hackers, highlighting a concerning potential for AI misuse.
Google’s threat intelligence department released a paper titled Adversarial Misuse of Generative AI, revealing how threat actors have approached their artificial intelligence chatbot Gemini.
According to Google, threat actors attempted to jailbreak the AI using prompts. In addition, government-backed advanced persistent threat (APT) groups have tried using Gemini to assist them in malicious endeavors.
Google reports unsuccessful attempts to jailbreak Gemini
Google said while threat actors had attempted to jailbreak Gemini, the company saw no advanced attempts in this attack vector.
According to Google, hackers only used basic measures like rephrasing or repeatedly sending the same prompt. Google said the attempts were unsuccessful.
AI jailbreaks are prompt injection attacks that aim to get an AI model to perform tasks that it had been prohibited from doing. This includes leaking sensitive information or providing unsafe content.
Google said that in one instance, an APT actor used publicly available prompts to trick Gemini into performing malicious coding tasks. However, Google said the attempt was unsuccessful as Gemini provided a safety-filtered response.
How government-backed threat actors used Gemini
In addition to low-effort jailbreak attempts, Google reported how government-backed APTs have approached Gemini.
Google said these attackers attempted to use Gemini to assist in their malicious activities. This included information gathering on their targets, researching publicly known vulnerabilities and coding and scripting tasks. In addition, Google said there have been attempts to enable post-compromise activities like defense evasion.
Google reported that Iran-based APT actors focused on using AI in crafting phishing campaigns. They also used the AI model to conduct recon on defense experts and organizations. The APT actors in Iran also used AI to generate cybersecurity content.
Meanwhile, China’s APT actors have used Gemini to troubleshoot code, scripting and development. In addition, they used AI to research how to obtain deeper access to their target networks.
APT actors in North Korea have also used Gemini for different phases of their attack lifecycle, from research to development. The report said:
“They also used Gemini to research topics of strategic interest to the North Korean government, such as the South Korean military and cryptocurrency.”
In 2024, North Korean hackers stole $1.3 billion in digital assets, according to Chainalysis.
