Fighting AI-powered voice scams

Dane Sherrets at HackerOne describes the troubling rise of AI voice scams and how to defend against them

 

Generative AI (GenAI) is revolutionising modern software solutions thanks to its powerful capabilities, wide-ranging integration and universal accessibility. However, while the majority of people and organisations are using GenAI for the good, this isn’t the case for everyone.  Unfortunately, it is also lowering the barrier to entry for cyber-criminals, enabling them to exploit cyber-security vulnerabilities without the need for extensive technical expertise, particularly in coding. 

 

Furthermore, the hasty integration of AI-powered features, combined with the lack of understanding about their potential security vulnerabilities, has opened many businesses up to new attack surfaces, negating much of the recent headway businesses have made in cyber-security defence.

 

A new generation of digital threat

Perhaps one of the biggest threats from GenAI is the way it lends itself to social engineering attacks like never before. Armed with AI-powered capabilities, phishing content can now mimic human communication styles with alarming accuracy, making the task of spotting phishing scams increasingly difficult.

 

In addition, hackers can now leverage new tools to fashion deep fakes and other scams that are remarkably hard to differentiate from genuine interactions. In fact, according to a recent study by McAfee, one in four people has already experienced some kind of AI voice scam, with 77% of victims saying they lost money as a result.

 

The troubling rise in deep fakes has also been gaining more media attention over the past few months, from celebrities being cloned for influencer ads to a Hong Kong employee being duped into paying £20m. In another highlighted case, a CEO was tricked into transferring $243,000 by a fraudulent call in which the scammer used voice cloning. This perpetuated threat will only grow more sinister as the technology continues to evolve, alongside the experience and ability of cyber-criminals. 

 

How does AI voice cloning work?

AI voice cloning tools are typically trained on large datasets of audio recordings and can quickly learn to mimic different voices and accents. Therefore, a CEO’s voice can be replicated with minimal audio recordings, even without publicly available clips. 

 

McAfee’s study found that as little as three seconds of audio was enough to produce an 85% match, but with more investment and effort, it’s possible to significantly increase the accuracy. By training the data models more extensively, McAfee researchers were able to achieve a 95% voice match based on just a small number of audio files.

 

Once an accurate sample has been produced, it is uploaded into a specialised AI tool. The user can then input text or even speak directly into the tool, prompting it to generate audio in the cloned voice, complete with possible inflexions or pauses to enhance realism. The upload process takes less than five minutes with some of the free and open-source tools available today, signifying a surprisingly low barrier to conducting such sophisticated scams.

 

When scammers target victims, they use voice notes or live voice calls to create as much urgency and confusion as possible to increase their chances of success. This is not a specialised skill but something anyone can do with some time and effort, which is what makes it so dangerous.

 

Criminal applications beyond voice cloning

Sadly, voice cloning isn’t the only application criminals are using GenAI for. It is also highly effective at ‘humanising’ traditional phishing emails. In fact, the National Cyber Security Centre (NCSC) recently said people would soon struggle to identify phishing messages due to the sophistication of AI tools. According to the NCSC, this will “almost certainly” increase the volume of cyber-attacks and heighten their impact over the next two years.

 

It goes without saying that many leading GenAI tools have security built in to prevent their use in criminal activity, but there are already numerous examples of these being bypassed. Cyber-criminals often attempt to “jailbreak” LLMs, for example, to bypass rules around producing harmful content.

 

Alternatively, but quite similarly, criminals can also perform prompt injection in an attempt to quietly insert malicious data or instructions into AI models. No matter the avenue, this form of threat requires organisations to adopt an innovative approach to both safety and security. 

 

How can businesses defend themselves?

In the face of these increasingly sophisticated attacks, businesses must conduct constant testing and implement robust security measures focused on the specific threat GenAI poses. In particular, security leaders need to ensure it plays a major role in user training and awareness initiatives. Like all phishing attacks, awareness and vigilance remain the most effective form of defence. Below are some of the top courses of action employees should take if they suspect voice cloning:

• Listen for anomalies and unusual speech patterns such as atypical pauses or phrasing, and unusual background noises. Voice cloning may not perfectly mimic natural speech patterns.
• Question contents of requests and suspicious communications with personal knowledge a scammer wouldn’t know.
• Confirm the identities of callers with a safe word or phrase known only to close contacts for an immediate authenticity check against a scammer.
• Mind their digital footprint. Every public voice clip expands an individual’s "audio attack surface”, potentially giving scammers material to clone their voice. Consequently, every employee should consider the necessity and visibility of their online audio interactions.

As the popularity of GenAI continued to grow at an exponential rate, it was only a matter of time before criminal applications for the technology came to the fore. However, like all social engineering-based cyber-attacks, forewarned is forearmed. Training employees to stay vigilant and question anything out of the ordinary will go a long way towards keeping sensitive information safe.

 

For those wanting even greater peace of mind, engaging with the ethical hacking community offers an ideal way to pressure test existing security, as well as identify robust new defences as/where needed.

 

---

 

Dane Sherrets is Senior Solutions Architect at HackerOne

 

Main image courtesy of iStockPhoto.com and B4LLS