Guarding the future against AI hallucinations

Peter Chestna at Checkmarx outlines how to mitigate AI risks in software development

 

As artificial intelligence (AI) continues to revolutionise software development, developers now often rely on generative AI models at all stages from designing to coding and testing. Large language models (LLMs) like ChatGPT are even beginning to replace community forums such as Stack Overflow as a resource for coding challenges. 

 

However, this convenience comes with significant security risks, including AI ‘hallucinations’ that present false information as fact. Additionally, AI can disrupt established developer workflows, and LLMs often lack secure coding practices. This results in insecure code overwhelming AppSec teams who are already facing challenges as a result of under-resourcing and more complex code environments. 

 

As developers increasingly depend on AI, addressing these security challenges is vital to ensure robust software development.

 

The risks of AI hallucinations  

These risks are already taking effect and recent research has highlighted that AI can produce false information, or ’hallucinate’, about non-existent open-source software packages. If ChatGPT suggests a code library or package that doesn’t exist, a hacker could create a malicious package with the same name and distribute it to unsuspecting developers who trust AI’s recommendations, introducing a new threat into the software supply chain.

 

The impact of using malicious code goes beyond potentially severe compromise of software; it can have an impact on developer and organisational reputation, as well.

 

AI hallucinations present cyber-criminals with an opportunity to replace traditional techniques of introducing malicious code like ‘typosquatting’ and ‘masquerading’, which are more detectable. The security experiment also found that researchers used common questions from Stack Overflow and fed them to ChatGPT, observing whether an AI model recommended non-existent packages. 

 

The results show malicious packages based on AI recommendations have already been detected on widely used package installers such as PyPI and npm. This indicates that the threat is not merely theoretical, but present and growing. 

 

Also, the practice of "copy-paste-execute" becomes particularly risky in this context. Developers might copy a command provided by AI and execute it without verification, assuming its suggestions are safe. This introduces significant security risks, reinforcing the need for developers to provide oversight and verify AI-generated recommendations to proactively protect their software projects. 

 

Security to combat AI-generated threats 

AI is not the first major technological development to have added pressure to developers. Recent advancements in IoT have brought their own risks, but there is an added sense of urgency with AI given the speed with which new models are rolling out. 

 

This pace of change - and the pressure to develop new applications in shorter timeframes - often conflicts with the stringent requirements of robust application security. Developers focus on creating feature-rich, user-friendly applications, while security teams are dedicated to ensuring these applications are secure and free from vulnerabilities.

 

Industry experts, such as the Open Web Application Security Project (OWASP), advocate for secure coding practices and rigorous code reviews, especially when using third-party code. Verifying the origin of the code, authenticating developers through signed commits and packages, and sourcing open-source artefacts from reliable vendors are crucial steps.

 

As approximately 60% of software vulnerabilities are identified during the coding, building, or testing phases, a comprehensive security approach is essential.

 

Whilst we need to give full focus to the risks that AI can introduce in software development, AI models are also bringing huge benefits to teams by detecting and remediating security vulnerabilities.

 

This not only expedites remediation but also bridges knowledge gaps between AppSec teams and developers, fostering a more inclusive and effective approach to application security. Embedding AppSec protocols directly into AI-driven code generation workflows streamlines development efforts, ensuring that security is a foundational element of the software development lifecycle (SDLC).  

 

Enhancing developer security awareness

When mitigating the risks of AI in software development, it’s not only about the technology deployed: there are processes that can be introduced to ensure that development and security teams are fully aligned.

 

For example, establishing clear key performance indicators (KPIs) to include metrics like the number of vulnerabilities detected in early scans, and the time taken to mitigate them, provides a structured approach which not only tracks security progress but also helps evaluate improvements over time, and against business objectives.

 

Skills also need to keep pace with the rate of change to ensure that developers are able to integrate AI models into their coding practices. There is now a growing number of resources and courses which developers can access to bridge these gaps so that they can reap the productivity benefits of AI into their workflows. 

 

However, security needs to be at the heart of their practices. Embedding security guidance within developers’ familiar Integrated Development Environments (IDEs) can significantly boost their adherence to security practices. By providing security insights and tools within the environments that developers use daily, organisations can enhance the developer experience and build awareness of the importance of security practices. 

 

This awareness is critical: when developers understand the impact of security measures on the company’s overall success, they are likely to be more engaged with security protocols. Giving developers greater responsibility and a voice in AppSec decisions, particularly given the changes brought by AI, also helps to cement their role as stakeholders in building secure development processes. 

 

AI is transforming the way software is developed and delivering enormous benefits in terms of speed and effectiveness. However, this integration comes with significant challenges and these require human oversight.

 

It’s still a nascent field for software developers and by understanding these risks and implementing comprehensive security measures throughout the entire development process businesses can safeguard their projects and maintain the integrity of their software.

 

Ultimately, speed and efficiency should not be sacrificed for security.  

 

---

 

Peter Chestna is NA CISO at Checkmarx 

 

Main image courtesy of iStockPhoto.com