When AI turns against us

Giles Inkson at NetSPI describes how to defend against AI-powered cyber-attacks by combining the power of AI with the skills and creativity of people

 

AI Assistants, MCPs, and agents are commonly used by defenders as well as threat actors. Anthropic recently disclosed that its Claude chatbot had been exploited to carry out large-scale theft and extortion, even helping suspected North Korean agents fraudulently land jobs at US-based technology companies.

 

These types of activities can assist with attack lifecycle, automating reconnaissance, harvesting credentials, penetrating networks, crafting extortion demands and the full attack lifecycle, according to Anthropic’s data.

 

This underscores how cyber-criminals are no longer just experimenting with AI; they are actively weaponising its abilities, reshaping the trajectory and scale of attacks. The UK’s National Cyber Security Centre has echoed these concerns, warning that AI is making cyber-intrusions more effective, efficient and frequent.

 

With high-profile incidents such as the Jaguar Land Rover attacks underscoring the stakes, organisations need to understand how AI is reshaping the threat landscape and rethink their defences accordingly. AI is not only supercharging cyber-crime but also shifting the balance of power between attackers and defenders.

 

Across the criminal ecosystem, AI is being weaponised at speed, while legal and defensive frameworks struggle to keep pace. Security leaders need adapt by their own usages of AI technologies, without losing sight of the human expertise and oversight that remain essential.

 

The new criminal playbook

AI is reshaping cyber-crime at every level of sophistication. At the low end, commodity groups are using freely available tools to automate phishing, reconnaissance and basic malware generation. This “democratisation effect” lowers the barrier to entry, giving unsophisticated attackers capabilities once reserved for advanced operations and raising the baseline threat level for every organisation.

 

More sophisticated criminal enterprises are going further, developing custom AI-based applications to optimise attacks. Ransomware operators, for example, can use AI to analyse public data and identify victims most likely to pay, while cryptocurrency thieves can apply pattern recognition to target vulnerable wallets and time their attacks for maximum impact.

 

Nation-state actors: AI at geopolitical scale

At the highest end of the spectrum, nation-state actors are pushing AI into strategic domains. With resources to build custom models trained on specific industries, they generate disinformation at scale, manage convincing digital personas and automate the collection of intelligence to identify high-value targets.

 

In January 2024, the World Economic Forum labelled disinformation as the biggest short-term risk of AI globally warning it could undermine democracy and fuel social unrest. These risks are not theoretical. A 2023 deepfake audio clip of the Mayor of London Sadiq Khan sparked “serious disorder” ahead of Armistice Day. And recent research uncovered the use of AI tools to craft fake South Korean ID cards as part of a phishing attempt by North Korea-sponsored actors.

 

Such operations also make attribution harder. Language models strip away linguistic fingerprints while automated infrastructure generation conceals technical signatures. Combined with uneven international regulation, where some nations focus on defensive AI while others pursue offensive capabilities, these developments create dangerous imbalances that advanced actors exploit.

 

Bolstering defence: human-machine teaming

The most effective defence strategy is a hybrid of human and machine. This is where AI handles routine tasks whilst humans provide strategic direction and ethical oversight. This human-machine approach maximises both efficiency and effectiveness.

 

Organisations must invest in both offensive and defensive AI capabilities whilst maintaining human expertise for strategic decision-making. Success requires realistic expectations, careful implementation planning and continuous adaptation as technology evolves. The balance between automation benefits and implementation complexity determines overall programme success.

 

Industry collaboration: closing the gaps

No single organisation can solve this problem in isolation. Cross-industry collaboration, supported by professional standards and regulatory frameworks, is essential to managing AI risk. Information sharing about AI-enabled threats must increase, but competitive pressures and regulatory constraints often limit cooperation opportunities. Industry bodies should lead the way in establishing frameworks that enable collaboration without compromising proprietary advantage.

 

Standards development is equally critical. Professional organisations are beginning to establish AI-specific security practices, but these must balance innovation with risk mitigation and remain adaptable as the technology evolves. Certification schemes may help, though rapid advances risk making them obsolete quickly.

 

The AI arms race

AI is already reshaping the balance of power between attackers and defenders. Criminals are innovating faster than many organisations can adapt, exploiting both off-the-shelf tools and custom-built models to operate at unprecedented scale and speed. Nation-states are pushing the envelope even further, combining offensive AI with information warfare and intelligence operations.

 

But AI also gives defenders new capabilities. Organisations that combine human expertise with machine efficiency will not only withstand these changes but potentially gain a strategic advantage. The challenge lies in avoiding extremes, neither over-relying on automation nor dismissing AI as hype.

 

The security community is entering an arms race where both attackers and defenders are evolving with AI. The organisations that thrive will be those that invest in resilient, adaptive programs, collaborate across industries and keep human expertise firmly at the core of their security strategy.

 

---

 

Giles Inkson is Director of Red Team & Adversary Simulation at NetSPI

 

Main image courtesy of iStockPhoto.com and gremlin