Balancing AI innovation and security

LLMs struggle to generate consistently secure code. Security-skilled developers can help ensure secure AI-generated code while optimising performance. Pieter Danhieux at Secure Code Warrior explores a CISO’s role in AI-powered coding

 

Development teams must gain better control over the large language models used in writing software code to prevent those AI models, which have brought undeniable benefits, from becoming a runaway train in terms of lax security protocols. 

 

Software developers were quick to see AI’s advantages - a little more than a half-year after ChatGPT made its initial splash in November 2022, a GitHub survey found that its Copilot coding assistant wrote 82 billion lines of code in its first year. 

 

The drawback of utilising LLMs to write code is the risk they pose to security. And while vulnerabilities have always been a concern in software, the rapid evolution of cloud services has only increased the demand for code. If LLMs are relied upon to meet that demand without having proper oversight of the security and quality of the code carefully checked, the consequences could be significant.

 

In our own experiments using LLMs to complete secure coding challenges, we often see error rates from 10% up to 60%, with the most prominent models averaging around 20-25%. It’s critical to note that this is a controlled situation in which we’re purposely prompting it in relation to security problems. If your prompt is not security-centric with the correct asks, your chances of success will be worse.

 

In terms of vulnerability classes, some are definitely easier for LLMs to navigate than others. They tend to score well on superficial, well-documented patterns such as SQL and other injection vulnerabilities, but have difficulties with more subjective, flexible issues like resource releasing, insufficient logging, and misconfigured permissions.

 

This situation underscores a looming crisis for the cybersecurity industry, while also creating an opportunity. It is clear that CISOs and security leaders need a robust plan for integrating AI coding tools safely in order to safeguard their systems and data, avoid the repercussions of a major breach and stay in compliance with an increasing amount of regulations. Such a plan will enable companies to leverage the considerable advantages AI coding tools offer while ensuring there is a reliable process in place for fast, productive and secure software development.

 

Capitalising on those benefits starts with a focus on risk reduction at the developer level.

 

Recognising the limitations of AI

It’s not like code was flawless before AI showed up. Human software engineers make their share of mistakes, too. A study by Coralogix found that developers create, on average, 70 bugs per 1,000 lines of code, with 15 of those bugs making their way into production systems. As a result, 75% of developers’ time is spent on debugging, since fixing bugs takes 30 times longer than writing a line of code. 

 

At a glance, AI models actually improve those numbers. Nearly 76% of respondents to a Snyk survey claimed that, overall, AI code is more secure than code created by humans.

 

But it’s far from perfect—56.4% said AI does introduce coding issues either sometimes or frequently. And considering the sheer volume of code AI creates—along with 80% of developers using AI to bypass AI code security policies—this threatens to put significant buggy code into the software ecosystem at breakneck speed. 

 

Relying too heavily on AI coding tools in their current form is risky, as models may have difficulty delivering consistent and reliable results, especially at the enterprise level. 

 

An example of LLM’s shortcomings can be found in how the models are ill-equipped to stay current with changes in functionalities. As Andrea Valenzuela, a software developer and data scientist at CERN, points out, LLMs are trained on a snapshot of code and documentation taken at a specific point in time. But APIs and other interfaces, for example, change often. Since LLMs aren’t updated in real-time, they often remain unaware of emerging security risks, potentially leading to the use of vulnerable code.

 

Although LLMs can be trained to write code, they are trained to predict the next line of code based on what’s come before. However, training them to write code optimised for specific business functions—or certain hardware or software environments—is an extremely challenging task.

 

Other potential vulnerabilities resulting from LLM-generated code include data poisoning used to manipulate machine learning models, the theft of LLM models, which can result in the creation of counterfeit models, adversarial inputs that trick LLMs into producing faulty output and biases present in training data that manifest within its output. Cross-site scripting is another potential vulnerability resulting from AI-generated code. In fact, LLM code flaws are common enough that the OWASP Foundation has developed a wiki just for the top 10 most critical LLM vulnerabilities.

 

Setting the stage for secure AI coding 

Organisations aren’t going to abandon AI over these concerns. In fact, the trend is moving strongly towards increased adoption. However, they must recognise that AI models can’t be relied upon to consistently generate secure and optimally functional code.

 

CISOs need to equip their organisations’ foundations to apply security and oversight to LLMs to ensure that they get maximum benefits from AI-generated code while applying strict security controls to the process.

 

Another factor to consider is the decision-making process: Who will determine which AI agent should be used? As we’ve observed, there are several LLMs available, each with its own strengths and shortcomings, and in terms of coding, one may prove to be more accurate than another.

 

Ultimately, highly regulated enterprise environments like the financial services sector will likely operate with a central decision, but in more flexible environments, such as the tech sector, the choice may be left up to individual developers, which will vastly increase the risk and governance variables in the SDLC. 

 

Among the steps they can take: 

• AI governance: Implement a framework to establish safe and ethical practices and policies for using AI and machine learning. A governance team should involve key stakeholders from across the enterprise, including IT, data science, legal, compliance and business.
• Regulatory legislation: Companies should stay informed about governmental efforts to restrict AI use. The EU AI Act is the first regulatory framework that specifically applies to AI. The United States doesn’t yet have legislation directly addressing AI, but the White House Executive Order from October 2023 does set standards for safety, transparency, and security. 
• Upskilling and reskilling: Secure code is at the core of cybersecurity, and improving the security and quality of AI-based code starts with developers. Organisations must ensure developers are equipped with verified skills and knowledge to apply secure coding best practices in the code they write and in checking the work of code generated by LLMs. 

Teams need precision skills development as part of a comprehensive, data-driven program designed for integrating security in both the code creation process and throughout the entire software development lifecycle (SDLC). A developer-focussed security program can boost productivity, streamline the SLDC workflow and spur innovation, while also making software more secure and reliable.

 

A key component of that program is ensuring that upskilling efforts are taking hold with developers. Utilising a platform that offers measurement of a security learning program’s effectiveness while also identifying top performers and those who need extra help is essential. It can also provide benchmarks that identify areas that need to be addressed by the learning program and assess the organisation’s performance relative to the rest of the industry.

 

Critical importance of secure code

Senior leaders must recognise the critical importance of secure code and the need to continuously enable developers to be thoroughly versed in safe coding practices. Creating an environment that allows LLMs to generate code under the supervision of security-aware developers can enable organisations to improve productivity, while concentrating on risk mitigation at both the developer and supplementary technology levels.

 

---

 

Pieter Danhieux is CEO and Co-Founder at Secure Code Warrior

 

Main image courtesy of iStockPhoto.com and monsitj