How ChatGPT upended email

Dr Kiri Addison and Evonne Lee at Mimecast describe the double-edged sword of AI 

 

ChatGPT has transformed the way most people, without much technical or AI knowledge, perceive and think about AI. It has become much more accessible, less intimidating and now a tool that many are beginning to rely on.

 

There is no doubt its capabilities have quickly made it a go-to tool for individuals and businesses to streamline or even take over many everyday tasks, and with its impressive understanding of human prompts, it’s only becoming more sophisticated.

 

An easy-to-use platform that simplifies your daily tasks and crafts flawless emails sounds great, doesn’t it? But amidst the convenience, there’s a significant risk to your business: phishing attacks.

 

Email phishing is an increasing problem for business across all sizes and sectors, and generative AI systems like ChatGPT are partly to blame for creating huge increases in malicious phishing emails. Essentially, cyber-criminals are using generative AI tools (such as ChatGPT) to help write sophisticated, convincing and extremely targeted business email compromise (BEC) attacks and other phishing messages.

 

How has this happened? Due to their usage, these AI systems now have access to substantial amounts of data to generate accurate insights and content, which makes them valuable for enhancing productivity, automating tasks, and streamlining ideas.

 

However, if not properly regulated or protected, the same systems can inadvertently expose sensitive information or intellectual property (IP), leading to serious cyber-breaches and reputational damage.

 

Exploring the threat prevalence 

We’ve established that AI-generated content is becoming more widespread and it’s becoming increasingly challenging to distinguish between human and machine-written text. However, one key giveaway? The use of complex words and sentence structures that AI models tend to favour.

 

Researchers who analysed 14 million scientific papers from 2010 to 2024 found an unmistakable rise in the use of specific "AI-style" words after late 2022, when AI tools became more widespread. Words like "delves" surged 25-fold in 2024, while others such as "showcasing," "underscores," and "crucial" also saw significant increases, reflecting the influence of AI in shaping today’s writing styles.

 

Following the release of our recent Threat Intelligence report, we wanted to explore AI’s role in phishing attacks in more detail. It appeared no one could quantify just how widespread AI-generated phishing emails are. This sparked important questions, how common are these threats, and can we track them?

 

The data science team at Mimecast took on the challenge to help, by building a detection engine to determine if a message is human or AI-generated based on a mixture of current, historical and synthetic AI-generated emails. It’s no coincidence that the research indicated a clear point in time when we start observing an increasing trend in AI-generated emails correlating with the release of ChatGPT.  

 

Blurring the lines between human and machine 

Our detection engine has learned to identify specific characteristics that distinguish human-written emails from AI-generated ones. By analysing over 20,000 emails, along with synthetic data generated by models like GPT-4, we found that certain phrases—such as “delve deeper into this” or overly casual greetings like “hello!” from senders who don’t typically use such language, often signals a phishing attempt.

 

In total, 30,000 emails were sampled between January 2022 and June 2024. The findings are alarming, 7.8%, or 2,330 emails, were identified as AI-written. It is important to note that the model was not looking to identify malicious AI-written emails, but rather to estimate the pervasiveness of AI. But the real insight lies in the trend - a sharp rise in AI-generated content and a corresponding decline in human-written emails.

 

While the exact reasons behind the increase remain unclear, one possibility is that non-native English speakers are using AI tools to refine their writing. As AI continues to reshape communication, the implications for both personal and professional correspondence are significant and concerning for business security.

 

Manual phishing investigations

These findings emphasise the ongoing need for manual phishing investigations as a vital layer of defence, particularly when suspicious messages are flagged by end users. No solution can catch all AI-generated email attacks, so the ability to recognise phishing emails remains vital for protecting your organisation from cyber-threats.

 

While cyber-defences are becoming increasingly sophisticated, human intervention through risk management and contextual awareness remains invaluable in identifying threats that might sometimes bypass machine learning filters. Threat intelligence researchers play a key role in dissecting the language of flagged messages, identifying the key markers that align with the latest intelligence on phishing tactics.

 

By comparing these linguistic patterns with known threat behaviours, security teams can more effectively detect phishing attacks, reducing both response times and organisational risk. However, to stay ahead of AI-generated threats, the key threat indicators must be update continually to match the rapid growth of large language models and the arrival of new datasets.

 

Cyber-security should be an ongoing conversation. To truly safeguard our digital environment, security professionals need to foster a culture of security awareness. This involves regularly engaging users and strengthening training so that security becomes a core value within the organisation from board to employee. When a workforce understand that security is a business priority, they’ll be more likely to report suspicious emails and security concerns.

 

The main takeaway should this, as cyber-criminals evolve their tactics, defences must evolve too, ensuring that both humans and automated systems work in tandem to protect organisations.

 

---

 

Dr Kiri Addison is Mimecast’s Senior Manager, Product Management and Evonne Lee is Machine Learning Engineer at Mimecast

 

Main image courtesy of iStockPhoto.com and D3Damon