AI and the rise of malicious GPTs

Mike Britton at Abnormal AI argues that AI-powered defence must evolve to keep pace 

 

Cyber-criminals are always quick to appropriate the latest technology for malevolent ends, and AI has been no exception. Just as businesses have embraced generative AI tools to bolster productivity, threat actors have adopted them with equal enthusiasm to power their attacks.  

 

As AI-fuelled cyber-crime becomes more scalable and effective, the question facing organisations is no longer if they’ll be targeted, but how prepared they’ll be when it happens. 

 

Lowering the barrier for cyber-criminals 

Threat actors wasted little time jumping on the AI bandwagon, exploiting popular large language model (LLM) tools for tasks such as researching targets and writing convincing social engineering copy.  

 

While most reputable GenAI tools have safeguards to prevent obviously malicious and criminal requests, our research found that popular platforms including ChatGPT, DeepSeek, Claude and Gemini can be manipulated into criminal activity with the right prompts.  

 

Additionally, many other adversarial groups, not content to skirt around the ethical boundaries of legitimate tools, have developed their own malicious GPTs that have no constraints and are purpose-built for crime.  

 

WormGPT was one of the first notable examples of this trend, built on an open-source GPT-J language model to assist with attacks such as business email compromise. The tool has led to multiple spinoff variants including WolfGPT and Escape GPT. 

 

FraudGPT is another even more finely tuned AI tool seeing widespread use in cyber-criminal circles. Experimenting with the tool, we found it could rapidly create polished and convincing phishing copy even with minimal and incomplete prompts. Its ability to incorporate context means it functions as an ideal “starter kit”, filling in the gaps between other criminal assets like zero-days and malware guides. 

 

In late 2024, Abnormal AI researchers uncovered GhostGPT, one of the most recent and dangerous additions to the lineup. Marketed openly on Telegram, it promises fast responses, a no-logs policy, and ready-made access without the technical hurdles required to jailbreak a model or host your own. Users can pay a small fee, ask sensitive questions, and receive unfiltered, harmful outputs in seconds. 

 

What’s most concerning is the accessibility. With tools like GhostGPT, attackers don’t need coding expertise or insider knowledge. Anyone with an internet connection and a motive can launch attacks that were once the domain of skilled adversaries. 

 

GhostGPT offers a revealing snapshot of the way these tools are pushing AI-driven cyber-crime into the mainstream. 

 

The real impact of malicious AI 

AI misuse is no longer a theoretical risk - it’s playing out in real-world attacks with alarming consequences.   

 

In one high-profile case, a multinational company in Hong Kong lost $25 million after an employee was tricked into a video call featuring deepfake versions of the CFO and other executives. AI-generated avatars, scripted by fraudsters, were convincing enough to bypass traditional warning signs. 

 

Meanwhile, researchers have also shown how LLMs can be used to create polymorphic malware code that mutates with each iteration to evade detection. Even long-running scams like “pig butchering” are evolving, with generative AI now used to craft emotionally manipulative messages at scale, posing as romantic partners or financial advisors. 

 

These examples are just the beginning. The combination of automation and personalisation offered by malicious AI is enabling threat groups to build attack chains that are faster, more scalable, and far harder to detect. The old models of defence simply weren’t built for this kind of threat. 

 

Why traditional defences are failing 

Despite years of investment in perimeter defences, secure email gateways, and rule-based filters, most traditional tools are ill-equipped to handle AI-generated threats. These systems rely on static signatures and keyword-based detection methods that can be easily outmanoeuvred by the seemingly authentic, human-like language produced by malicious GPTs. 

 

What makes these attacks especially dangerous is their subtlety. Messages are often free of the obvious red flags common in human-made content. No typos, no suspicious links, no strange formatting. Instead, they mimic legitimate communication styles with uncanny accuracy, slipping past technical controls and landing directly in inboxes - where they rely on human error to do the rest. 

 

A dynamic defence for dynamic threats 

AI threats are best countered with AI defences, with behavioural AI being one of the most effective options. Unlike traditional defences that rely on static rules or predefined threat signatures, behavioural AI analyses patterns in user activity, communication style, and context, flagging the subtle anomalies that suggest something is off. 

 

Behavioural AI helps organisations adapt in real time. It uses machine learning to build a baseline of normal activity and respond dynamically to anything unusual.  For example, suppose an executive who typically communicates with short, direct messages suddenly sends a long, urgent request for a wire transfer, especially to an unfamiliar recipient. In that case, behavioural AI can identify and flag the deviation before damage is done. It doesn’t matter how well the message is written or how legitimate it looks. What matters is that it’s out of character. 

 

This context-aware detection is crucial when dealing with AI-generated attacks, which often bypass traditional filters by design.  

 

Proactive adaptation: the only defence 

The rise of malicious GPTs is just the tip of the iceberg. We are already seeing more mature threat groups adopting advanced strategies, including end-to-end automation of the attack chain, multimodal deception, and custom threat models.  

 

Agility is essential against the dynamic threat posed by these evolving threats. If attackers are using intelligent tools to bypass defences, defenders need intelligent tools of their own - built to anticipate, interpret, and act in the face of uncertainty. 

 

The future of cyber-crime is automated, scalable, and AI-powered. Defences need to be the same.  

 

---

 

Mike Britton is CIO at Abnormal AI 

 

Main image courtesy of iStockPhoto.com and blackdovfx