Taking over AI chatbots

Phil Robinson at Prism Infosec describes how AI-driven chatbots can be exploited and the steps that can be taken to reduce this risk

 

Chatbots or conversational AI, are computer programs designed to simulate human conversation and interaction using natural language processing (NLP) and AI. They are used to handle tasks from answering questions and providing information to offering support and can perform a variety of more complex tasks to such as integrating with calendars and scheduling appointments, responding to emails, or even writing code.

 

As such, they’ve become indispensable in many business environments and that’s also made them a prime target for attackers.

 

Chat injection sees the information flow of the chatbot effectively hijacked and tricked into doing something it shouldn’t by either directly targeting backend systems to access insecure functions and data stores linked to the language model or indirectly via external sources like websites, pdf documents or audio files. 

 

Just as with SQL injection or command injection, whereby an attacker can target the user input to manipulate the system’s output in order to compromise the confidentiality, integrity or availability of systems and data, chat injection can be used to compromise users, reveal sensitive information, influence critical decisions, or bypass safeguards.

 

AI injection attacks can see the chatbot manipulated by an attacker such as through prompt engineering, manipulating the data feed in terms of knowledge bases or APIs, or through injecting malicious code through the chatbot interface.

 

Compromising chatbots IRL

We decided to take a look at just how vulnerable these chatbots could be and so set about looking for issues in open source WordPress AI chatbot plugins. The results were startling and should serve as a warning to any business looking to deploy this technology to thoroughly test both existing AI chatbots they have in service and any they are planning to deploy.

 

The first chatbot we found issues with was AI Engine which plugin integrates with AI models from OpenAI, Anthropic, and Google, including GPT-4o, GPT-4 and Vision. It has its own internal APIs, allowing it to be integrate AI capabilities to WordPress to enhance the functionality of other plugins. It’s easy to set up via an Open AI account, simply requiring the user to create and API key and insert the plugin settings. 

 

In version 2.4.3 which had been downloaded 2.6m times and was used in 70,000 active installations, we found two issues which have subsequently been disclosed via WPScan, recognised as common vulnerabilities and exposures (CVEs) and remediated. 

 

The first issue pertained to remote code execution (RCE) whereby AI Engine was shown to be susceptible to log poisoning. Because the plugin fails to validate the file extension, anyone with admin access can change the log filetypes. This means that error messages can be manipulated and the change is then echoed in the log file and seen as being legitimate code by the web server, allowing RCE.

 

We also found a second CVE that enabled SQL injection that could performed via the admin. Both of these issues have since been addressed in release 2.4.8.

 

As these attacks required administrator access, it could be argued that the potential for them to be exploited were much less likely. In contrast, another chatbot we studied was susceptible to attacks without the need for authenticated access. 

 

Why authenticated access is a must

SmartSearch WP uses Open AI’s ChatGPT for customer interactions and to perform smart search and is described as an advanced Chatbot that can be customised to include only data that is relevant to the website, improving the accuracy of its responses, and is also multi-lingual. Version 2.4.2 which had been downloaded 2,000 times and had over 30 active installations was found to have two CVEs.

 

The first issue was that the chatbot did not properly check data before using it in an SQL statement, which meant it was potentially susceptible to SQL injection by unauthenticated users when they were submitting messages to the chatbot. But we also found a second issue that would allow an unauthenticated stored cross site scripting (XSS) attack. 

 

The attack started out with a self-XSS which meant that a payload that we had submitted to the chatbot would only be accessible to us but we found we could propagate it to become stored in the administrative ‘chat logs’. This means it could later be executed against an administrator viewing those logs at which point the XSS payload would execute within their web browser and user session.

 

We tailored the payload to steal the ChatGPT API key from the chatbot’s settings and were able to forward the key on to our attacker domain. Thankfully, both these SmartSearch issues have been addressed in version 2.4.5.

 

How to prevent chatbot takeover

These CVEs illustrate just how vulnerable AI chatbots can be to abuse and takeover. Interacting with a compromised chatbot can result in phishing scams, system compromises or disclosing personal information, for example, so it’s vital that these chatbots are routinely tested and updated.

 

There are a number of steps the business can take to mitigate these risks. At the development stage, input validation and message sanitisation of messages should be used to minimise the impact of potentially malicious inputs. Rate limiting, such as through throttling user requests and implementing automated lockouts, can also help deter rapid fire injection attempts or automated tools/scripts. 

 

Additionally, user authentication and verification along with IP and device monitoring can help deter anonymous online attackers by requiring some sort of identification before using the service. The concept of least privilege principle should also be applied to ensure that the chatbot can only access the data it needs. 

 

From a user’s perspective, it’s advisable to be cautious when sharing sensitive information with chat bots to prevent data theft. Incorporate human oversight for critical operations to add a layer of validation which will act as a safeguard against unintended or potentially malicious actions.

 

And finally, any systems that the chatbot integrates with should be secured to minimise impact in the event of a compromise. 

 

---

 

Phil Robinson is Principal Security Consultant at Prism Infosec

 

Main image courtesy of iStockPhoto.com and da-kuk