Why human oversight of AI systems remains essential

Ben Large at Cybit explains how human oversight of AI systems can be made effective

 

Businesses need to step up and protect their environment from rising cyber-threats. Whilst the evolution of Artificial Intelligence (AI) has benefitted companies in multiple ways, it has also empowered cyber-criminals and increased the risk of threats to businesses. AI lowers the barrier for cyber-criminals as the tech can be leveraged to identify system vulnerabilities and research targets.

 

According to GOV.UK, over 50% of UK businesses identified cyber-security breaches in 2024, and 44% of those businesses were victims of cyber-crime. Companies, regardless of their size, should be implementing robust firewalls that can detect both known and emerging threats, along with application security solutions that mitigate risks, as well as support and enable growth.

 

Growing threat of AI

As AI and machine learning skills continue to develop, it’s no surprise that there will be an increase in AI-based attacks. This includes AI-generated phishing emails that are harder to detect, along with AI’s ability to adapt in real-time to hack security systems.

 

This is concerning not only because AI tools are accessible to anyone, including threat actors. Also, due to the lack of technical skills required, AI is becoming easier to use. One example of this is Malware-as-a-Service (MaaS), where cyber-criminals offer their tools and related services to others for a fee. This enables unskilled attackers to access and customise software to execute advanced attacks. 

 

As cyber-attack tactics evolve, a market for AIMAS (AI Malware-as-a-Service) may develop. This would mean that AI can be sold and distributed as a tool for malware, enabling AI bots to automatically generate content and identify additional methods to scam people. 

 

Ultimately, as AI tools become more powerful, cyber-criminals will increasingly weaponise them. This highlights the importance of human oversight, especially today when the threat of attacks is higher than ever. Humans can reaffirm ethical reasoning and have a wider context over cultural norms and human circumstances compared to AI, which flags behaviours as ‘suspicious’ with a lack of knowledge.

 

Importance of human oversight

AI cannot be held legally or morally responsible without human oversight. Humans remain accountable for how AI tools are used, with human oversight ensuring humans are present in the process to take responsibility and accountability. It’s critical for identifying false positives and negatives as well as monitoring outcomes. Humans play a key role in determining bias and in adapting models to ensure legal compliance.

 

AI systems are commonly used within companies and infrastructures. It’s important to manage the data that they are trained on to ensure total accuracy. In surveillance systems like CCTV, the quality and control of data is crucial because if the data is incomplete, poorly labelled and unchecked, the AI will learn incorrect or biased patterns.

 

The biases that exist in data are then used to train AI. Without human oversight, these data issues could lead to misidentifying individuals or unfairly targeting specific groups. Therefore, keeping human oversight is important to inspect data to remove bias and confirm its validity and ethical responsibility.

 

How human oversight can be made effective

Facilitating effective human oversight is essential to ensure operational efficiency; however, it’s not always a simple task. To start, understanding who is behind an AI model is crucial, as it represents the goals of the people or organisations building the model. It’s also important to understand who’s funding the models, as whoever is paying for the model may have influence over what it does and how it behaves. For example, a private company may create an AI model to maximise profits.

 

Due to its computing power, engineering time and data processing, AI models can be extremely expensive to create. To make human oversight more effective and enable more ethical and reliable AI models, the training data should be validated to ensure there are no errors or bias before training, especially when dealing with large datasets. However, this can be costly and time-consuming.

 

Interestingly, geopolitics often has a significant influence on AI models as the inputs can be biased. Countries may feed in propaganda or one-sided data; similarly, the models could be trained on data from Western perspectives, excluding other cultural factors.

 

This affects the output validity as biased outputs will make the AI inaccurate and unreliable overall. As it stands, the EU AI Act regulates how AI is built and used within the European Union; however, the EU has no legal power to enforce those rules outside of Europe. Therefore, other countries can utilise AI models however they like, regardless of regulations, highlighting the difficulties that can come with controlling AI’s effectiveness and ethical impact on a more global scale.

 

Continuous learning for ongoing threats

As computer power gets faster, the computation of the models using larger data sets will get easier. There will likely be a point in the future where computer power will allow any individual to create an AI model to learn malware, dating back years, as well as all the computer vulnerabilities up to that day, and actively attack systems.

 

To overcome risks and protect IT environments, cyber-security strategies must prioritise continuous learning for continuous threats. It’s important to adapt in real-time and take each risk and attempted breach as a learning opportunity. Ensuring that employees stay educated on the latest threats, as well as analysing and checking for biased data on models, is a priority. Additionally, ensuring that human oversight is not lost and remains effective will be crucial now and in the future. 

 

Doing the minimum just isn’t enough. This is where companies must remain switched on and up to date regarding their security measures to protect their environments. Machine learning, leveraging AI, and human oversight are essential to ensure that, as threats evolve, people do as well. 

 

---

 

Ben Large is UK Technical Lead at Cybit

 

Main image courtesy of iStockPhoto.com and Devrimb